25 May
2021
25 May
'21
4:23 p.m.
Hi,
I am still searching for a reason why these two values differ so much.
I am currently deleting a giant amount of orphan objects (43mio, most
of them under 64kb), but the difference get larger instead of smaller.
This was the state two days ago:
[root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' |
awk '{ print $2 }' | tr -d , | paste -sd+ - | bc
175977343264
[root@s3db1 ~]# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY
UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
...
eu-central-1.rgw.buckets.data 766 TiB 134632397 0 403897191 0
0 0 1076480853 45 TiB 532045864 551 TiB 0 B 0 B
...
total_objects 135866676
[root@s3db1 ~]# ceph df...
eu-central-1.rgw.buckets.data 11 2048 253 TiB 134.63M 766 TiB
90.32 27 TiB
And this is todays state:
[root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' |
awk '{ print $2 }' | tr -d , | paste -sd+ - | bc
177144806812
[root@s3db1 ~]# rados df
...
eu-central-1.rgw.buckets.data 786 TiB 120025590 0 360076770
...
total_objects 121261889
[root@s3db1 ~]# ceph df
...
eu-central-1.rgw.buckets.data 11 2048 260 TiB 120.02M 786 TiB
92.59 21 TiB
I would love to free up the missing 80TB :)
Any suggestions?
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
25 May
25 May
4:39 p.m.
Hi,
On 25 May 2021, at 10:23, Boris Behrens
<bb(a)kervyn.de> wrote:
I am still searching for a reason why these two values differ so much.
I am currently deleting a giant amount of orphan objects (43mio, most
of them under 64kb), but the difference get larger instead of smaller.
When user trough API make a delete, objects just marks as deleted, then
ceph-radosgw gc perform actual delete, you can see queue via `radosgw-admin gc list`
I think you can speedup process via rgw_gc_ options.
Cheers,
k
5:07 p.m.
Am Di., 25. Mai 2021 um 09:39 Uhr schrieb Konstantin Shalygin <k0ste(a)k0ste.ru>ru>:
Hi,
On 25 May 2021, at 10:23, Boris Behrens <bb(a)kervyn.de> wrote:
I am still searching for a reason why these two values differ so much.
I am currently deleting a giant amount of orphan objects (43mio, most
of them under 64kb), but the difference get larger instead of smaller.
When user trough API make a delete, objects just marks as deleted, then
ceph-radosgw gc perform actual delete, you can see queue via `radosgw-admin gc list`
I think you can speedup process via rgw_gc_ options.
Cheers,
k
Hi K,
I thought about the GC, but it doesn't look like this is the issue:
[root@s3db1 ~]# radosgw-admin gc list --include-all | grep oid | wc -l
563598
[root@s3db1 ~]# radosgw-admin gc list | grep oid | wc -l
43768
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
9:41 p.m.
Am Di., 25. Mai 2021 um 09:23 Uhr schrieb Boris Behrens <bb(a)kervyn.de>de>:
Hi,
I am still searching for a reason why these two values differ so much.
I am currently deleting a giant amount of orphan objects (43mio, most
of them under 64kb), but the difference get larger instead of smaller.
This was the state two days ago:
As Konstatin mentioned, maybe it was the GC, but I just processes all
objects (with --include-all), but the situation did not change.
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
[root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' |
awk '{ print $2 }' | tr -d , | paste -sd+ - | bc
175977343264
[root@s3db1 ~]# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY
UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
...
eu-central-1.rgw.buckets.data 766 TiB 134632397 0 403897191 0
0 0 1076480853 45 TiB 532045864 551 TiB 0 B 0 B
...
total_objects 135866676
[root@s3db1 ~]# ceph df...
eu-central-1.rgw.buckets.data 11 2048 253 TiB 134.63M 766 TiB
90.32 27 TiB
And this is todays state:
[root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' |
awk '{ print $2 }' | tr -d , | paste -sd+ - | bc
177144806812
[root@s3db1 ~]# rados df
...
eu-central-1.rgw.buckets.data 786 TiB 120025590 0 360076770
...
total_objects 121261889
[root@s3db1 ~]# ceph df
...
eu-central-1.rgw.buckets.data 11 2048 260 TiB 120.02M 786 TiB
92.59 21 TiB
I would love to free up the missing 80TB :)
Any suggestions?
26 May
26 May
3:27 p.m.
The more files I delete, the more space is used.
How can this be?
Am Di., 25. Mai 2021 um 14:41 Uhr schrieb Boris Behrens <bb(a)kervyn.de>de>:
Am Di., 25. Mai 2021 um 09:23 Uhr schrieb Boris Behrens <bb(a)kervyn.de>de>:
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
Hi,
I am still searching for a reason why these two values differ so much.
I am currently deleting a giant amount of orphan objects (43mio, most
of them under 64kb), but the difference get larger instead of smaller.
This was the state two days ago:
As Konstatin mentioned, maybe it was the GC, but I just processes all
objects (with --include-all), but the situation did not change.
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
[root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' |
awk '{ print $2 }' | tr -d , | paste -sd+ - | bc
175977343264
[root@s3db1 ~]# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY
UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
...
eu-central-1.rgw.buckets.data 766 TiB 134632397 0 403897191 0
0 0 1076480853 45 TiB 532045864 551 TiB 0 B 0 B
...
total_objects 135866676
[root@s3db1 ~]# ceph df...
eu-central-1.rgw.buckets.data 11 2048 253 TiB 134.63M 766 TiB
90.32 27 TiB
And this is todays state:
[root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' |
awk '{ print $2 }' | tr -d , | paste -sd+ - | bc
177144806812
[root@s3db1 ~]# rados df
...
eu-central-1.rgw.buckets.data 786 TiB 120025590 0 360076770
...
total_objects 121261889
[root@s3db1 ~]# ceph df
...
eu-central-1.rgw.buckets.data 11 2048 260 TiB 120.02M 786 TiB
92.59 21 TiB
I would love to free up the missing 80TB :)
Any suggestions?
1069
days inactive
1070
days old
4 comments
2 participants
participants (2)
-
Boris Behrens -
Konstantin Shalygin