Im prettty sure that you got the same issue than we already reported :
https://tracker.ceph.com/issues/43756
Garbage and garbage stored into our OSD without be able to cleanup wasting a lot of
space.
As you can see its solved in the new versions but... the last versión didnt have any
"scrub" or similar system to fix the garbage generated in the past versions.
As result , even big companies got their RGW plattform with tons of TB wasted.
Eric, Is there a way to ask you to develop(RGW Team) a system to clean our rgw clusters
like rgw bucket scrub?
I talked with Cbodley and he explained how to it manually but the process is extremely
complex.
We already calculated that at least a 25% of our rgw cluster is garbage (100TB), and our
options right now:
- Deploy a new cluster a move rgw Users one by one with their buckets with an external
copy, hopping in the last nautilus version this not happen again (Not usefull option and
not transparent)
- Buy disk and disk waiting for a solution as External Tool (No sure to continue this
way)
- Hire external developers with knowleage of ceph and create a private tool for that.
(Developers with Ceph Core/Rgw knowleage will be no easy to find)
Here: ceph version 14.2.8
-----Mensaje original-----
De: Eric Ivancich <ivancich(a)redhat.com>
Enviado el: martes, 28 de abril de 2020 18:39
Para: Katarzyna Myrek <katarzyna(a)myrek.pl>
CC: ceph-users(a)ceph.io
Asunto: [ceph-users] Re: RGW and the orphans
Hi Katarzyna,
Incomplete multipart uploads are not considered orphans.
With respect to the 404s…. Which version of ceph are you running? What tooling are you
using to list and cancel? Can you provide a console transcript of the listing and
cancelling?
Thanks,
Eric
--
J. Eric Ivancich
he / him / his
Red Hat Storage
Ann Arbor, Michigan, USA
On Apr 28, 2020, at 2:57 AM, Katarzyna Myrek
<katarzyna(a)myrek.pl> wrote:
Hi all
I am afraid that there is even more thrash available - running
rgw-orphan-list does not find everything. Like I still have broken
multiparts -> when I do s3cmd multipart I get a list of
"pending/interrupted multiparts". When I try to cancel such multipart
I get 404.
Does anyone have a method for cleanup of such things? Or even a list
of tasks which should be run regularly on clusters with rgw ?
Kind regards / Pozdrawiam,
Katarzyna Myrek
wt., 21 kwi 2020 o 09:57 Janne Johansson <icepic.dz(a)gmail.com> napisał(a):
Den tis 21 apr. 2020 kl 07:29 skrev Eric Ivancich <ivancich(a)redhat.com>om>:
Please be certain to read the associated docs in both:
doc/radosgw/orphans.rst
doc/man/8/rgw-orphan-list.rst
so you understand the limitations and potential pitfalls. Generally this tool will be a
precursor to a large delete job, so understanding what’s going on is important.
I look forward to your report! And please feel free to post additional questions in this
forum.
Where are those?
https://github.com/ceph/ceph/tree/master/doc/man/8
https://github.com/ceph/ceph/tree/master/doc/radosgw
don't seem to contain them in master. Nor in nautilus branch or octopus.
This whole issue feels weird, rgw (or its users) produces dead
fragments of mulitparts, orphans and whatnot that needs cleaning up sooner or later and
the info we get is that the old cleaner isn't meant to be used, it hasn't worked
for a long while, there is no fixed version, perhaps there is a script somewhere with
caveats. This (slightly frustrated) issue is of course on top of "bi trim"
"bilog trim"
"mdlog trim"
"usage trim"
"datalog trim"
"sync error trim"
"gc process"
"reshard stale-instances rm"
that we rgw admins are supposed to know when to run, how often, what their quirks are and
so on.
'Docs' for rgw means "datalog trim" --help says "trims the
datalog", and the long version on the web would be "this operation trims the
datalog" or something that doesn't add anything more.
--
"Grumpy cat was an optimist"
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io