summarized radosgw size_kb_actual vs pool stored value doesn't add up
Hi, I am still searching for a reason why these two values differ so much. I am currently deleting a giant amount of orphan objects (43mio, most of them under 64kb), but the difference get larger instead of smaller. This was the state two days ago:
[root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' | awk '{ print $2 }' | tr -d , | paste -sd+ - | bc 175977343264
[root@s3db1 ~]# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR ... eu-central-1.rgw.buckets.data 766 TiB 134632397 0 403897191 0 0 0 1076480853 45 TiB 532045864 551 TiB 0 B 0 B ... total_objects 135866676
[root@s3db1 ~]# ceph df... eu-central-1.rgw.buckets.data 11 2048 253 TiB 134.63M 766 TiB 90.32 27 TiB
And this is todays state:
[root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' | awk '{ print $2 }' | tr -d , | paste -sd+ - | bc 177144806812
[root@s3db1 ~]# rados df ... eu-central-1.rgw.buckets.data 786 TiB 120025590 0 360076770 ... total_objects 121261889
[root@s3db1 ~]# ceph df ... eu-central-1.rgw.buckets.data 11 2048 260 TiB 120.02M 786 TiB 92.59 21 TiB
I would love to free up the missing 80TB :) Any suggestions? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Hi,
On 25 May 2021, at 10:23, Boris Behrens <bb@kervyn.de> wrote:
I am still searching for a reason why these two values differ so much.
I am currently deleting a giant amount of orphan objects (43mio, most of them under 64kb), but the difference get larger instead of smaller.
When user trough API make a delete, objects just marks as deleted, then ceph-radosgw gc perform actual delete, you can see queue via `radosgw-admin gc list` I think you can speedup process via rgw_gc_ options. Cheers, k
Am Di., 25. Mai 2021 um 09:39 Uhr schrieb Konstantin Shalygin <k0ste@k0ste.ru>:
Hi,
On 25 May 2021, at 10:23, Boris Behrens <bb@kervyn.de> wrote:
I am still searching for a reason why these two values differ so much.
I am currently deleting a giant amount of orphan objects (43mio, most of them under 64kb), but the difference get larger instead of smaller.
When user trough API make a delete, objects just marks as deleted, then ceph-radosgw gc perform actual delete, you can see queue via `radosgw-admin gc list` I think you can speedup process via rgw_gc_ options.
Cheers, k
Hi K, I thought about the GC, but it doesn't look like this is the issue:
[root@s3db1 ~]# radosgw-admin gc list --include-all | grep oid | wc -l 563598 [root@s3db1 ~]# radosgw-admin gc list | grep oid | wc -l 43768
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Am Di., 25. Mai 2021 um 09:23 Uhr schrieb Boris Behrens <bb@kervyn.de>:
Hi, I am still searching for a reason why these two values differ so much.
I am currently deleting a giant amount of orphan objects (43mio, most of them under 64kb), but the difference get larger instead of smaller.
This was the state two days ago:
[root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' | awk '{ print $2 }' | tr -d , | paste -sd+ - | bc 175977343264
[root@s3db1 ~]# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR ... eu-central-1.rgw.buckets.data 766 TiB 134632397 0 403897191 0 0 0 1076480853 45 TiB 532045864 551 TiB 0 B 0 B ... total_objects 135866676
[root@s3db1 ~]# ceph df... eu-central-1.rgw.buckets.data 11 2048 253 TiB 134.63M 766 TiB 90.32 27 TiB
And this is todays state:
[root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' | awk '{ print $2 }' | tr -d , | paste -sd+ - | bc 177144806812
[root@s3db1 ~]# rados df ... eu-central-1.rgw.buckets.data 786 TiB 120025590 0 360076770 ... total_objects 121261889
[root@s3db1 ~]# ceph df ... eu-central-1.rgw.buckets.data 11 2048 260 TiB 120.02M 786 TiB 92.59 21 TiB
I would love to free up the missing 80TB :) Any suggestions?
As Konstatin mentioned, maybe it was the GC, but I just processes all objects (with --include-all), but the situation did not change. -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
The more files I delete, the more space is used. How can this be? Am Di., 25. Mai 2021 um 14:41 Uhr schrieb Boris Behrens <bb@kervyn.de>:
Am Di., 25. Mai 2021 um 09:23 Uhr schrieb Boris Behrens <bb@kervyn.de>:
Hi, I am still searching for a reason why these two values differ so much.
I am currently deleting a giant amount of orphan objects (43mio, most of them under 64kb), but the difference get larger instead of smaller.
This was the state two days ago:
[root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' | awk '{ print $2 }' | tr -d , | paste -sd+ - | bc 175977343264
[root@s3db1 ~]# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR ... eu-central-1.rgw.buckets.data 766 TiB 134632397 0 403897191 0 0 0 1076480853 45 TiB 532045864 551 TiB 0 B 0 B ... total_objects 135866676
[root@s3db1 ~]# ceph df... eu-central-1.rgw.buckets.data 11 2048 253 TiB 134.63M 766 TiB 90.32 27 TiB
And this is todays state:
[root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' | awk '{ print $2 }' | tr -d , | paste -sd+ - | bc 177144806812
[root@s3db1 ~]# rados df ... eu-central-1.rgw.buckets.data 786 TiB 120025590 0 360076770 ... total_objects 121261889
[root@s3db1 ~]# ceph df ... eu-central-1.rgw.buckets.data 11 2048 260 TiB 120.02M 786 TiB 92.59 21 TiB
I would love to free up the missing 80TB :) Any suggestions?
As Konstatin mentioned, maybe it was the GC, but I just processes all objects (with --include-all), but the situation did not change.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
participants (2)
-
Boris Behrens
-
Konstantin Shalygin