14 Mar
2023
14 Mar
'23
5:34 p.m.
Hi everyone, I’ve got a quick question regarding one of our RadosGW bucket.
This bucket is used to store docker registries, and the total amount of
data we use is supposed to be 4.5Tb BUT it looks like ceph told us we
rather use ~53Tb of data.
One interesting thing is, this bucket seems to shard for unknown reason as
it is supposed to be disabled by default, but even taking that into account
we’re not supposed to see such a massive amount of additional data isn’t it?
Here is the bucket stats of it:
https://paste.opendev.org/show/bdWFRvNFtxyHnbPfXWu9/
14 Mar
14 Mar
5:49 p.m.
On Tue, Mar 14, 2023 at 06:34:54PM +0100, Gaël THEROND wrote:
Hi everyone, I’ve got a quick question regarding one
of our RadosGW bucket.
This bucket is used to store docker registries, and the total amount of
data we use is supposed to be 4.5Tb BUT it looks like ceph told us we
rather use ~53Tb of data.
One interesting thing is, this bucket seems to shard for unknown reason as
it is supposed to be disabled by default, but even taking that into account
we’re not supposed to see such a massive amount of additional data isn’t it?
Here is the bucket stats of it:
https://paste.opendev.org/show/bdWFRvNFtxyHnbPfXWu9/
At a glance, is versioning
enabled?
And if so, are you pruning old versions?
Please share "radosgw-admin metadata get" for the bucket &
bucket-instance.
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robbat2(a)gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
5:59 p.m.
Versioning wasn’t enabled, at least not explicitly and for the
documentation it isn’t enabled by default.
Using nautilus.
I’ll get all the required missing information on tomorrow morning, thanks
for the help!
Is there a way to tell CEPH to delete versions that aren’t current used one
with radosgw-admin?
If not I’ll use the rest api no worries.
Le mar. 14 mars 2023 à 18:49, Robin H. Johnson <robbat2(a)gentoo.org> a
écrit :
On Tue, Mar 14, 2023 at 06:34:54PM +0100, Gaël THEROND
wrote:
Hi everyone, I’ve got a quick question regarding
one of our RadosGW
bucket.
This bucket is used to store docker registries, and the total amount of
data we use is supposed to be 4.5Tb BUT it looks like ceph told us we
rather use ~53Tb of data.
One interesting thing is, this bucket seems to shard for unknown reason
as
it is supposed to be disabled by default, but
even taking that into
account
we’re not supposed to see such a massive amount
of additional data isn’t
it?
Here is the bucket stats of it:
https://paste.opendev.org/show/bdWFRvNFtxyHnbPfXWu9/
At a glance, is versioning
enabled?
And if so, are you pruning old versions?
Please share "radosgw-admin metadata get" for the bucket &
bucket-instance.
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robbat2(a)gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
6:07 p.m.
On Tue, Mar 14, 2023 at 06:59:51PM +0100, Gaël THEROND wrote:
Versioning wasn’t enabled, at least not explicitly and
for the
documentation it isn’t enabled by default.
Using nautilus.
I’ll get all the required missing information on tomorrow morning, thanks
for the help!
Is there a way to tell CEPH to delete versions that aren’t current used one
with radosgw-admin?
If not I’ll use the rest api no worries.
Nope, s3 API only.
You should also check for incomplete multiparts. For that, I recommend
using AWSCLI or boto directly. Specifically not s3cmd, because s3cmd
doesn't respect the <IsTruncated> flag properly.
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robbat2(a)gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
6:18 p.m.
Alright,
Seems something is odd out there, if I do a radosgw-admin metadata list
I’ve got the following list:
[
”bucket”,
”bucket.instance”,
”otp”,
”user”
]
BUT
When I try a radosgw-admin metadata get bucket or bucket.instance it
complain with the following error:
ERROR: can’t get key: (22) Invalid argument
Ok, fine for the api, I’ll deal with the s3 api.
Even if a radosgw-admin bucket flush version —keep-current or something
similar would be much appreciated xD
Le mar. 14 mars 2023 à 19:07, Robin H. Johnson <robbat2(a)gentoo.org> a
écrit :
On Tue, Mar 14, 2023 at 06:59:51PM +0100, Gaël THEROND
wrote:
Versioning wasn’t enabled, at least not
explicitly and for the
documentation it isn’t enabled by default.
Using nautilus.
I’ll get all the required missing information on tomorrow morning, thanks
for the help!
Is there a way to tell CEPH to delete versions that aren’t current used
one
with radosgw-admin?
If not I’ll use the rest api no worries.
Nope, s3 API only.
You should also check for incomplete multiparts. For that, I recommend
using AWSCLI or boto directly. Specifically not s3cmd, because s3cmd
doesn't respect the <IsTruncated> flag properly.
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robbat2(a)gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
8:44 p.m.
Hi,
I found the documentation for metadata get to be unhelpful for what
syntax to use. I eventually found that it's this:
radosgw-admin metadata get bucket:{bucket_name}
or
radosgw-admin metadata get bucket.instance:{bucket_name}:{instance_id}
Hopefully that helps you or someone else struggling with this.
Rich
On Wed, 15 Mar 2023 at 07:18, Gaël THEROND <gael.therond(a)bitswalk.com> wrote:
>
> Alright,
> Seems something is odd out there, if I do a radosgw-admin metadata list
>
> I’ve got the following list:
>
> [
> ”bucket”,
> ”bucket.instance”,
> ”otp”,
> ”user”
> ]
>
> BUT
>
> When I try a radosgw-admin metadata get bucket or bucket.instance it
> complain with the following error:
>
> ERROR: can’t get key: (22) Invalid argument
>
> Ok, fine for the api, I’ll deal with the s3 api.
>
> Even if a radosgw-admin bucket flush version —keep-current or something
> similar would be much appreciated xD
>
> Le mar. 14 mars 2023 à 19:07, Robin H. Johnson <robbat2(a)gentoo.org> a
> écrit :
>
> > On Tue, Mar 14, 2023 at 06:59:51PM +0100, Gaël THEROND wrote:
> > > Versioning wasn’t enabled, at least not explicitly and for the
> > > documentation it isn’t enabled by default.
> > >
> > > Using nautilus.
> > >
> > > I’ll get all the required missing information on tomorrow morning, thanks
> > > for the help!
> > >
> > > Is there a way to tell CEPH to delete versions that aren’t current used
> > one
> > > with radosgw-admin?
> > >
> > > If not I’ll use the rest api no worries.
> > Nope, s3 API only.
> >
> > You should also check for incomplete multiparts. For that, I recommend
> > using AWSCLI or boto directly. Specifically not s3cmd, because s3cmd
> > doesn't respect the <IsTruncated> flag properly.
> >
> > --
> > Robin Hugh Johnson
> > Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
> > E-Mail : robbat2(a)gentoo.org
> > GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
> > GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
15 Mar
15 Mar
10:26 a.m.
Aaaah many many thanks for the information rich! It helped a lot!
Indeed the rados operations documentation lack a bunch of advanced commands
explanations.
I’ve finally find the exact and explicit command explanation in here for
the record: https://docs.ceph.com/en/latest/radosgw/layout/#metadata
Regarding my own metadata get bucket:
https://paste.opendev.org/show/bOI6gItMdDMvXSs2Swf5/
&
For the metadata get bucket.instance:
https://paste.opendev.org/show/bRPec1KViuklmNIjasU4/
I’ve also checked our min_alloc size, I’m using default 64K with EC of 2+1
which seems to match the data space amplification spreadsheet as a bunch of
our registry objects are 0 to 1780 octets size and ranging from there to
large object that fit the min_alloc size.
Here is an object detailed:
https://paste.opendev.org/show/bp7nvVPq5a9mbF6Ad3Jc/
So all in all we may well be in that case.
I still have a question about that, is there a way to optimize that a bit
as x10 is a bit huge even if we get way more available space, I’m wondering
if the fragmented free space could be retrieved or at least ceph count for
the real object size used and not the reserved size?
If anyone is having thoughts on the situation in here I’m all hears out if
it can improve and reduce the storage size usage.
Kind regards,
G.
Le mar. 14 mars 2023 à 21:44, Richard Bade <hitrich(a)gmail.com> a écrit :
Hi,
I found the documentation for metadata get to be unhelpful for what
syntax to use. I eventually found that it's this:
radosgw-admin metadata get bucket:{bucket_name}
or
radosgw-admin metadata get bucket.instance:{bucket_name}:{instance_id}
Hopefully that helps you or someone else struggling with this.
Rich
On Wed, 15 Mar 2023 at 07:18, Gaël THEROND <gael.therond(a)bitswalk.com>
wrote:
Alright,
Seems something is odd out there, if I do a radosgw-admin metadata list
I’ve got the following list:
[
”bucket”,
”bucket.instance”,
”otp”,
”user”
]
BUT
When I try a radosgw-admin metadata get bucket or bucket.instance it
complain with the following error:
ERROR: can’t get key: (22) Invalid argument
Ok, fine for the api, I’ll deal with the s3 api.
Even if a radosgw-admin bucket flush version —keep-current or something
similar would be much appreciated xD
Le mar. 14 mars 2023 à 19:07, Robin H. Johnson <robbat2(a)gentoo.org> a
écrit :
> On Tue, Mar 14, 2023 at 06:59:51PM +0100, Gaël THEROND wrote:
> > Versioning wasn’t enabled, at least not explicitly and for the
> > documentation it isn’t enabled by default.
> >
> > Using nautilus.
> >
> > I’ll get all the required missing information on tomorrow morning,
thanks
> > for the help!
> >
> > Is there a way to tell CEPH to delete versions that aren’t current
used
one
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io with radosgw-admin?
If not I’ll use the rest api no worries.
Nope, s3 API only.
You should also check for incomplete multiparts. For that, I recommend
using AWSCLI or boto directly. Specifically not s3cmd, because s3cmd
doesn't respect the <IsTruncated> flag properly.
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robbat2(a)gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
14 Mar
14 Mar
6:37 p.m.
Is it possible that you are storing object (chunks if EC) that are
smaller than the min_alloc size? This cheat sheet might help:
https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUTo…
Mark
On 3/14/23 12:34, Gaël THEROND wrote:
> Hi everyone, I’ve got a quick question regarding one of our RadosGW bucket.
>
> This bucket is used to store docker registries, and the total amount of
> data we use is supposed to be 4.5Tb BUT it looks like ceph told us we
> rather use ~53Tb of data.
>
> One interesting thing is, this bucket seems to shard for unknown reason as
> it is supposed to be disabled by default, but even taking that into account
> we’re not supposed to see such a massive amount of additional data isn’t it?
>
> Here is the bucket stats of it:
> https://paste.opendev.org/show/bdWFRvNFtxyHnbPfXWu9/
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:42 p.m.
Thanks a lot for this spreadsheet, I’ll check that on but I doubt we store
data smaller than the min_alloc size.
Yes we do use an EC pool type of 2+1 with failure_domain being at host
level.
Le mar. 14 mars 2023 à 19:38, Mark Nelson <mark.a.nelson(a)gmail.com> a
écrit :
Is it possible that you are storing object (chunks if
EC) that are
smaller than the min_alloc size? This cheat sheet might help:
https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUTo…
Mark
On 3/14/23 12:34, Gaël THEROND wrote:
Hi everyone, I’ve got a quick question regarding
one of our RadosGW
bucket.
This bucket is used to store docker registries, and the total amount of
data we use is supposed to be 4.5Tb BUT it looks like ceph told us we
rather use ~53Tb of data.
One interesting thing is, this bucket seems to shard for unknown reason
as
it is supposed to be disabled by default, but
even taking that into
account
we’re not supposed to see such a massive amount
of additional data isn’t
it?
Here is the bucket stats of it:
https://paste.opendev.org/show/bdWFRvNFtxyHnbPfXWu9/
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
414
days inactive
415
days old
8 comments
4 participants
participants (4)
-
Gaël THEROND -
Mark Nelson -
Richard Bade -
Robin H. Johnson