Hi
@Eric Ivancich my cluster has some history and trash gathered over the
years. Most (terabytes) is from
.
I was able to reproduce the problem on my LAB and it is for sure connected
with
. When you are on a version older
than 14.2.8 you would need to apply lifecycle policy which tries to abort
interrupted multiparts older than x days. And when the bucket index is
sharded then the broken/un-cancellable MPs are born.
To test it I can use s3cmd. My LAB cluster was upgraded to 14.2.8 to make
sure the new version does not do cleanup automagically.
Here is my procedure (I truncated my personal data):
*s3cmd --access_key= --secret_key= --host= --host-bucket= multipart
s3://kate-mp-issue*
*s3://kate-mp-issue/Initiated Path Id2020-04-06T07:48:55.323Z
s3://kate-mp-issue/bottest_20200406T074855.img
2~-9SKkHzGKXYX_zNdHNs_S8RY9hWjISS*
*s3cmd --access_key= --secret_key= --host= --host-bucket= abortmp
s3://kate-mp-issue/bottest_20200406T074855.img
2~-9SKkHzGKXYX_zNdHNs_S8RY9hWjISS*
*ERROR: S3 error: 404 (NoSuchUpload)*
*RGW logs:*
2020-04-29 07:24:23.126 7fb21b819700 1 ====== starting new request
req=0x7fb21b8128d0 =====
2020-04-29 07:24:23.126 7fb21b819700 1 ====== req done req=0x7fb21b8128d0
op status=0 http_status=200 latency=0s ======
2020-04-29 07:24:23.126 7fb21b819700 1 civetweb: 0x381a000: IP - -
[29/Apr/2020:07:24:22 +0000] "GET /kate-mp-issue/?location HTTP/1.1" 200
275 - -
2020-04-29 07:24:23.202 7fb21b819700 1 ====== starting new request
req=0x7fb21b8128d0 =====
2020-04-29 07:24:23.202 7fb21b819700 1 ====== req done req=0x7fb21b8128d0
op status=-2009 *http_status=404* latency=0s ======
2020-04-29 07:24:23.202 7fb21b819700 1 civetweb: 0x381a000: Ip - -
[29/Apr/2020:07:24:22 +0000] "DELETE
/kate-mp-issue/bottest_20200406T074855.img?uploadId=2~-9SKkHzGKXYX_zNdHNs_S8RY9hWjISS
HTTP/1.1" *404* 439 - -
So basically I need to remove those ghost entries from the list of
interrupted multiparts and clean up objects which are left overs.
As far as I understand I would need to go over every object in the pool
with `rados ls`, then compare the output with `radosgw-admin bi list (done
for every bucket)` and with new command `radosgw-admin radoslist` and
remove objects which are on 1 but not on 2 and 3. Plus clean up the
interrupted multipart list. Is that correct?
@EDH - Manuel Rios <mriosfer(a)easydatahost.com> Is that your method also?
I really need to clean up the terbaytes of the leftovers, because my prod
cluster is getting full. And now buying anything is not an option (harsh
times due to pandemy).
Kind regards / Pozdrawiam,
Katarzyna Myrek
Kind regards / Pozdrawiam,
Katarzyna Myrek
wt., 28 kwi 2020 o 19:45 EDH - Manuel Rios <mriosfer(a)easydatahost.com>
napisał(a):
Im prettty sure that you got the same issue than we
already reported :
https://tracker.ceph.com/issues/43756
Garbage and garbage stored into our OSD without be able to cleanup wasting
a lot of space.
As you can see its solved in the new versions but... the last versión
didnt have any "scrub" or similar system to fix the garbage generated in
the past versions.
As result , even big companies got their RGW plattform with tons of TB
wasted.
Eric, Is there a way to ask you to develop(RGW Team) a system to clean our
rgw clusters like rgw bucket scrub?
I talked with Cbodley and he explained how to it manually but the process
is extremely complex.
We already calculated that at least a 25% of our rgw cluster is garbage
(100TB), and our options right now:
- Deploy a new cluster a move rgw Users one by one with their buckets with
an external copy, hopping in the last nautilus version this not happen
again (Not usefull option and not transparent)
- Buy disk and disk waiting for a solution as External Tool (No sure to
continue this way)
- Hire external developers with knowleage of ceph and create a private
tool for that. (Developers with Ceph Core/Rgw knowleage will be no easy to
find)
Here: ceph version 14.2.8
-----Mensaje original-----
De: Eric Ivancich <ivancich(a)redhat.com>
Enviado el: martes, 28 de abril de 2020 18:39
Para: Katarzyna Myrek <katarzyna(a)myrek.pl>
CC: ceph-users(a)ceph.io
Asunto: [ceph-users] Re: RGW and the orphans
Hi Katarzyna,
Incomplete multipart uploads are not considered orphans.
With respect to the 404s…. Which version of ceph are you running? What
tooling are you using to list and cancel? Can you provide a console
transcript of the listing and cancelling?
Thanks,
Eric
--
J. Eric Ivancich
he / him / his
Red Hat Storage
Ann Arbor, Michigan, USA
On Apr 28, 2020, at 2:57 AM, Katarzyna Myrek
<katarzyna(a)myrek.pl> wrote:
Hi all
I am afraid that there is even more thrash available - running
rgw-orphan-list does not find everything. Like I still have broken
multiparts -> when I do s3cmd multipart I get a list of
"pending/interrupted multiparts". When I try to cancel such multipart
I get 404.
Does anyone have a method for cleanup of such things? Or even a list
of tasks which should be run regularly on clusters with rgw ?
Kind regards / Pozdrawiam,
Katarzyna Myrek
wt., 21 kwi 2020 o 09:57 Janne Johansson <icepic.dz(a)gmail.com>
napisał(a):
Den tis 21 apr. 2020 kl 07:29 skrev Eric Ivancich <ivancich(a)redhat.com
:
>>
>> Please be certain to read the associated docs in both:
>>
>> doc/radosgw/orphans.rst
>> doc/man/8/rgw-orphan-list.rst
>>
>> so you understand the limitations and potential pitfalls. Generally
this
tool will be a precursor to a large delete job, so understanding
what’s going on is important.
>> I look forward to your report! And please
feel free to post additional
questions in this forum.
>>
>
> Where are those?
>
https://github.com/ceph/ceph/tree/master/doc/man/8
>
https://github.com/ceph/ceph/tree/master/doc/radosgw
> don't seem to contain them in master. Nor in nautilus branch or octopus.
>
> This whole issue feels weird, rgw (or its users) produces dead
> fragments of mulitparts, orphans and whatnot that needs cleaning up
sooner or
later and the info we get is that the old cleaner isn't meant to
be used, it hasn't worked for a long while, there is no fixed version,
perhaps there is a script somewhere with caveats. This (slightly
frustrated) issue is of course on top of "bi trim"
> "bilog trim"
> "mdlog trim"
> "usage trim"
>
> "datalog trim"
>
> "sync error trim"
>
> "gc process"
>
> "reshard stale-instances rm"
>
>
>
> that we rgw admins are supposed to know when to run, how often, what
their
quirks are and so on.
>
>
> 'Docs' for rgw means "datalog trim" --help says "trims the
datalog",
and the long version on the web would be "this operation trims
the datalog"
or something that doesn't add anything more.
--
"Grumpy cat was an optimist"
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io