10x more used space than expected
Hi everyone, I’ve got a quick question regarding one of our RadosGW bucket. This bucket is used to store docker registries, and the total amount of data we use is supposed to be 4.5Tb BUT it looks like ceph told us we rather use ~53Tb of data. One interesting thing is, this bucket seems to shard for unknown reason as it is supposed to be disabled by default, but even taking that into account we’re not supposed to see such a massive amount of additional data isn’t it? Here is the bucket stats of it: https://paste.opendev.org/show/bdWFRvNFtxyHnbPfXWu9/
On Tue, Mar 14, 2023 at 06:34:54PM +0100, Gaël THEROND wrote:
Hi everyone, I’ve got a quick question regarding one of our RadosGW bucket.
This bucket is used to store docker registries, and the total amount of data we use is supposed to be 4.5Tb BUT it looks like ceph told us we rather use ~53Tb of data.
One interesting thing is, this bucket seems to shard for unknown reason as it is supposed to be disabled by default, but even taking that into account we’re not supposed to see such a massive amount of additional data isn’t it?
Here is the bucket stats of it: https://paste.opendev.org/show/bdWFRvNFtxyHnbPfXWu9/ At a glance, is versioning enabled?
And if so, are you pruning old versions? Please share "radosgw-admin metadata get" for the bucket & bucket-instance. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Versioning wasn’t enabled, at least not explicitly and for the documentation it isn’t enabled by default. Using nautilus. I’ll get all the required missing information on tomorrow morning, thanks for the help! Is there a way to tell CEPH to delete versions that aren’t current used one with radosgw-admin? If not I’ll use the rest api no worries. Le mar. 14 mars 2023 à 18:49, Robin H. Johnson <robbat2@gentoo.org> a écrit :
On Tue, Mar 14, 2023 at 06:34:54PM +0100, Gaël THEROND wrote:
Hi everyone, I’ve got a quick question regarding one of our RadosGW bucket.
This bucket is used to store docker registries, and the total amount of data we use is supposed to be 4.5Tb BUT it looks like ceph told us we rather use ~53Tb of data.
One interesting thing is, this bucket seems to shard for unknown reason as it is supposed to be disabled by default, but even taking that into account we’re not supposed to see such a massive amount of additional data isn’t it?
Here is the bucket stats of it: https://paste.opendev.org/show/bdWFRvNFtxyHnbPfXWu9/ At a glance, is versioning enabled?
And if so, are you pruning old versions?
Please share "radosgw-admin metadata get" for the bucket & bucket-instance.
-- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
On Tue, Mar 14, 2023 at 06:59:51PM +0100, Gaël THEROND wrote:
Versioning wasn’t enabled, at least not explicitly and for the documentation it isn’t enabled by default.
Using nautilus.
I’ll get all the required missing information on tomorrow morning, thanks for the help!
Is there a way to tell CEPH to delete versions that aren’t current used one with radosgw-admin?
If not I’ll use the rest api no worries. Nope, s3 API only.
You should also check for incomplete multiparts. For that, I recommend using AWSCLI or boto directly. Specifically not s3cmd, because s3cmd doesn't respect the <IsTruncated> flag properly. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Alright, Seems something is odd out there, if I do a radosgw-admin metadata list I’ve got the following list: [ ”bucket”, ”bucket.instance”, ”otp”, ”user” ] BUT When I try a radosgw-admin metadata get bucket or bucket.instance it complain with the following error: ERROR: can’t get key: (22) Invalid argument Ok, fine for the api, I’ll deal with the s3 api. Even if a radosgw-admin bucket flush version —keep-current or something similar would be much appreciated xD Le mar. 14 mars 2023 à 19:07, Robin H. Johnson <robbat2@gentoo.org> a écrit :
On Tue, Mar 14, 2023 at 06:59:51PM +0100, Gaël THEROND wrote:
Versioning wasn’t enabled, at least not explicitly and for the documentation it isn’t enabled by default.
Using nautilus.
I’ll get all the required missing information on tomorrow morning, thanks for the help!
Is there a way to tell CEPH to delete versions that aren’t current used one with radosgw-admin?
If not I’ll use the rest api no worries. Nope, s3 API only.
You should also check for incomplete multiparts. For that, I recommend using AWSCLI or boto directly. Specifically not s3cmd, because s3cmd doesn't respect the <IsTruncated> flag properly.
-- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Hi, I found the documentation for metadata get to be unhelpful for what syntax to use. I eventually found that it's this: radosgw-admin metadata get bucket:{bucket_name} or radosgw-admin metadata get bucket.instance:{bucket_name}:{instance_id} Hopefully that helps you or someone else struggling with this. Rich On Wed, 15 Mar 2023 at 07:18, Gaël THEROND <gael.therond@bitswalk.com> wrote:
Alright, Seems something is odd out there, if I do a radosgw-admin metadata list
I’ve got the following list:
[ ”bucket”, ”bucket.instance”, ”otp”, ”user” ]
BUT
When I try a radosgw-admin metadata get bucket or bucket.instance it complain with the following error:
ERROR: can’t get key: (22) Invalid argument
Ok, fine for the api, I’ll deal with the s3 api.
Even if a radosgw-admin bucket flush version —keep-current or something similar would be much appreciated xD
Le mar. 14 mars 2023 à 19:07, Robin H. Johnson <robbat2@gentoo.org> a écrit :
On Tue, Mar 14, 2023 at 06:59:51PM +0100, Gaël THEROND wrote:
Versioning wasn’t enabled, at least not explicitly and for the documentation it isn’t enabled by default.
Using nautilus.
I’ll get all the required missing information on tomorrow morning, thanks for the help!
Is there a way to tell CEPH to delete versions that aren’t current used one with radosgw-admin?
If not I’ll use the rest api no worries. Nope, s3 API only.
You should also check for incomplete multiparts. For that, I recommend using AWSCLI or boto directly. Specifically not s3cmd, because s3cmd doesn't respect the <IsTruncated> flag properly.
-- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Aaaah many many thanks for the information rich! It helped a lot! Indeed the rados operations documentation lack a bunch of advanced commands explanations. I’ve finally find the exact and explicit command explanation in here for the record: https://docs.ceph.com/en/latest/radosgw/layout/#metadata Regarding my own metadata get bucket: https://paste.opendev.org/show/bOI6gItMdDMvXSs2Swf5/ & For the metadata get bucket.instance: https://paste.opendev.org/show/bRPec1KViuklmNIjasU4/ I’ve also checked our min_alloc size, I’m using default 64K with EC of 2+1 which seems to match the data space amplification spreadsheet as a bunch of our registry objects are 0 to 1780 octets size and ranging from there to large object that fit the min_alloc size. Here is an object detailed: https://paste.opendev.org/show/bp7nvVPq5a9mbF6Ad3Jc/ So all in all we may well be in that case. I still have a question about that, is there a way to optimize that a bit as x10 is a bit huge even if we get way more available space, I’m wondering if the fragmented free space could be retrieved or at least ceph count for the real object size used and not the reserved size? If anyone is having thoughts on the situation in here I’m all hears out if it can improve and reduce the storage size usage. Kind regards, G. Le mar. 14 mars 2023 à 21:44, Richard Bade <hitrich@gmail.com> a écrit :
Hi, I found the documentation for metadata get to be unhelpful for what syntax to use. I eventually found that it's this: radosgw-admin metadata get bucket:{bucket_name} or radosgw-admin metadata get bucket.instance:{bucket_name}:{instance_id}
Hopefully that helps you or someone else struggling with this.
Rich
On Wed, 15 Mar 2023 at 07:18, Gaël THEROND <gael.therond@bitswalk.com> wrote:
Alright, Seems something is odd out there, if I do a radosgw-admin metadata list
I’ve got the following list:
[ ”bucket”, ”bucket.instance”, ”otp”, ”user” ]
BUT
When I try a radosgw-admin metadata get bucket or bucket.instance it complain with the following error:
ERROR: can’t get key: (22) Invalid argument
Ok, fine for the api, I’ll deal with the s3 api.
Even if a radosgw-admin bucket flush version —keep-current or something similar would be much appreciated xD
Le mar. 14 mars 2023 à 19:07, Robin H. Johnson <robbat2@gentoo.org> a écrit :
On Tue, Mar 14, 2023 at 06:59:51PM +0100, Gaël THEROND wrote:
Versioning wasn’t enabled, at least not explicitly and for the documentation it isn’t enabled by default.
Using nautilus.
I’ll get all the required missing information on tomorrow morning,
thanks
for the help!
Is there a way to tell CEPH to delete versions that aren’t current used one with radosgw-admin?
If not I’ll use the rest api no worries. Nope, s3 API only.
You should also check for incomplete multiparts. For that, I recommend using AWSCLI or boto directly. Specifically not s3cmd, because s3cmd doesn't respect the <IsTruncated> flag properly.
-- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Is it possible that you are storing object (chunks if EC) that are smaller than the min_alloc size? This cheat sheet might help: https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToP... Mark On 3/14/23 12:34, Gaël THEROND wrote:
Hi everyone, I’ve got a quick question regarding one of our RadosGW bucket.
This bucket is used to store docker registries, and the total amount of data we use is supposed to be 4.5Tb BUT it looks like ceph told us we rather use ~53Tb of data.
One interesting thing is, this bucket seems to shard for unknown reason as it is supposed to be disabled by default, but even taking that into account we’re not supposed to see such a massive amount of additional data isn’t it?
Here is the bucket stats of it: https://paste.opendev.org/show/bdWFRvNFtxyHnbPfXWu9/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks a lot for this spreadsheet, I’ll check that on but I doubt we store data smaller than the min_alloc size. Yes we do use an EC pool type of 2+1 with failure_domain being at host level. Le mar. 14 mars 2023 à 19:38, Mark Nelson <mark.a.nelson@gmail.com> a écrit :
Is it possible that you are storing object (chunks if EC) that are smaller than the min_alloc size? This cheat sheet might help:
https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToP...
Mark
On 3/14/23 12:34, Gaël THEROND wrote:
Hi everyone, I’ve got a quick question regarding one of our RadosGW bucket.
This bucket is used to store docker registries, and the total amount of data we use is supposed to be 4.5Tb BUT it looks like ceph told us we rather use ~53Tb of data.
One interesting thing is, this bucket seems to shard for unknown reason as it is supposed to be disabled by default, but even taking that into account we’re not supposed to see such a massive amount of additional data isn’t it?
Here is the bucket stats of it: https://paste.opendev.org/show/bdWFRvNFtxyHnbPfXWu9/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Gaël THEROND
-
Mark Nelson
-
Richard Bade
-
Robin H. Johnson