Hi Eric, I will try your tool this week on lab clusters. Will get back to you when I get the results. Kind regards / Pozdrawiam, Katarzyna Myrek pt., 17 kwi 2020 o 21:12 Eric Ivancich <ivancich(a)redhat.com> napisał(a): > > On Apr 17, 2020, at 9:38 AM, Katarzyna Myrek <katarzyna(a)myrek.pl> wrote: > > Hi Eric, > > Would it be possible to use it with an older cluster version (like > running new radosgw-admin in the container, connecting to the cluster > on 14.2.X)? > > Kind regards / Pozdrawiam, > Katarzyna Myrek > > > I did mention the nautilus backport PR in a separate reply (https://github.com/ceph/ceph/pull/34127). > > You can try the master version and see. To the best of my recollection the code porting did not involve any at-rest data structures. Instead it involved internal reorganization of the code. I suspect it would work, but if you try it, please report back what you find. Of course this is currently an experimental feature and care (e.g., sanity checking) should be taken before using the list produced to feed into a massive delete process. > > Eric > > -- > J. Eric Ivancich > > he / him / his > Red Hat Storage > Ann Arbor, Michigan, USA > > > > czw., 16 kwi 2020 o 19:58 EDH - Manuel Rios > <mriosfer(a)easydatahost.com> napisał(a): > > > Hi Eric, > > > > Are there any ETA for get those script backported maybe in 14.2.10? > > > > Regards > > Manuel > > > > > > De: Eric Ivancich <ivancich(a)redhat.com> > Enviado el: jueves, 16 de abril de 2020 19:05 > Para: Katarzyna Myrek <katarzyna(a)myrek.pl>pl>; EDH - Manuel Rios <mriosfer(a)easydatahost.com> > CC: ceph-users(a)ceph.io > Asunto: Re: [ceph-users] RGW and the orphans > > > > There is currently a PR for an “orphans list” capability. I’m currently working on the testing side to make sure it’s part of our teuthology suite. > > > > See: https://github.com/ceph/ceph/pull/34148 > > > > Eric > > > > > > On Apr 16, 2020, at 9:26 AM, Katarzyna Myrek <katarzyna(a)myrek.pl> wrote: > > > > Hi > > Thanks for the quick response. > > To be honest my cluster is getting full because of that trash and I am > at the point where I have to do the removal manually ;/. > > Kind regards / Pozdrawiam, > Katarzyna Myrek > > czw., 16 kwi 2020 o 13:09 EDH - Manuel Rios > <mriosfer(a)easydatahost.com> napisał(a): > > > Hi, > > From my experience orphans find didn't work since several releases ago, and command should be re-coded or deprecated because its not running. > > Im our cases it loops over generated shards until RGW daemon crash. > > Interested into this post, in our case orphans find takes more than 24 hours into start loop over shards, but never pass the shard 0 or 1. > > CEPH RGW devs, should provide any workaround script/ new tool or something to maintain our rgw clusters. Because with the last bugs all rgw cluster got a ton of trash, wasting resources and money. > > And manual cleaning is not trivial and easy. > > Waiting for more info, > > Manuel > > > -----Mensaje original----- > De: Katarzyna Myrek <katarzyna(a)myrek.pl> > Enviado el: jueves, 16 de abril de 2020 12:38 > Para: ceph-users(a)ceph.io > Asunto: [ceph-users] RGW and the orphans > > Hi > > Is there any new way to find and remove orphans from RGW pools on Nautilus? I have found info that "orphans find" is now deprecated? > > I can see that I have tons of orphans in one of our clusters. Was wondering how to safely remove them - make sure that they are really orphans. > Does anyone have a good method for that? > > My cluster mostly has orphans from multipart uploads. > > > Kind regards / Pozdrawiam, > Katarzyna Myrek > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > >

Den tis 21 apr. 2020 kl 07:29 skrev Eric Ivancich <ivancich(a)redhat.com>om>: Where are those? https://github.com/ceph/ceph/tree/master/doc/man/8 https://github.com/ceph/ceph/tree/master/doc/radosgw don't seem to contain them in master. Nor in nautilus branch or octopus. This whole issue feels weird, rgw (or its users) produces dead fragments of mulitparts, orphans and whatnot that needs cleaning up sooner or later and the info we get is that the old cleaner isn't meant to be used, it hasn't worked for a long while, there is no fixed version, perhaps there is a script somewhere with caveats. This (slightly frustrated) issue is of course on top of "bi trim" "bilog trim" "mdlog trim" "usage trim" "datalog trim" "sync error trim" "gc process" "reshard stale-instances rm" that we rgw admins are supposed to know when to run, how often, what their quirks are and so on. 'Docs' for rgw means "datalog trim" --help says "trims the datalog", and the long version on the web would be "this operation trims the datalog" or something that doesn't add anything more. -- "Grumpy cat was an optimist"

Hi all I am afraid that there is even more thrash available - running rgw-orphan-list does not find everything. Like I still have broken multiparts -> when I do s3cmd multipart I get a list of "pending/interrupted multiparts". When I try to cancel such multipart I get 404. Does anyone have a method for cleanup of such things? Or even a list of tasks which should be run regularly on clusters with rgw ? Kind regards / Pozdrawiam, Katarzyna Myrek wt., 21 kwi 2020 o 09:57 Janne Johansson <icepic.dz(a)gmail.com> napisał(a): > > Den tis 21 apr. 2020 kl 07:29 skrev Eric Ivancich <ivancich(a)redhat.com>om>: >> >> Please be certain to read the associated docs in both: >> >> doc/radosgw/orphans.rst >> doc/man/8/rgw-orphan-list.rst >> >> so you understand the limitations and potential pitfalls. Generally this tool will be a precursor to a large delete job, so understanding what’s going on is important. >> I look forward to your report! And please feel free to post additional questions in this forum. >> > > Where are those? > https://github.com/ceph/ceph/tree/master/doc/man/8 > https://github.com/ceph/ceph/tree/master/doc/radosgw > don't seem to contain them in master. Nor in nautilus branch or octopus. > > This whole issue feels weird, rgw (or its users) produces dead fragments of mulitparts, orphans and whatnot that needs cleaning up sooner or later and the info we get is that the old cleaner isn't meant to be used, it hasn't worked for a long while, there is no fixed version, perhaps there is a script somewhere with caveats. This (slightly frustrated) issue is of course on top of > "bi trim" > "bilog trim" > "mdlog trim" > "usage trim" > > "datalog trim" > > "sync error trim" > > "gc process" > > "reshard stale-instances rm" > > > > that we rgw admins are supposed to know when to run, how often, what their quirks are and so on. > > > 'Docs' for rgw means "datalog trim" --help says "trims the datalog", and the long version on the web would be "this operation trims the datalog" or something that doesn't add anything more. > > > > > -- > > "Grumpy cat was an optimist" >

Im prettty sure that you got the same issue than we already reported : https://tracker.ceph.com/issues/43756 Garbage and garbage stored into our OSD without be able to cleanup wasting a lot of space. As you can see its solved in the new versions but... the last versión didnt have any "scrub" or similar system to fix the garbage generated in the past versions. As result , even big companies got their RGW plattform with tons of TB wasted. Eric, Is there a way to ask you to develop(RGW Team) a system to clean our rgw clusters like rgw bucket scrub? I talked with Cbodley and he explained how to it manually but the process is extremely complex. We already calculated that at least a 25% of our rgw cluster is garbage (100TB), and our options right now: - Deploy a new cluster a move rgw Users one by one with their buckets with an external copy, hopping in the last nautilus version this not happen again (Not usefull option and not transparent) - Buy disk and disk waiting for a solution as External Tool (No sure to continue this way) - Hire external developers with knowleage of ceph and create a private tool for that. (Developers with Ceph Core/Rgw knowleage will be no easy to find) Here: ceph version 14.2.8 -----Mensaje original----- De: Eric Ivancich <ivancich(a)redhat.com> Enviado el: martes, 28 de abril de 2020 18:39 Para: Katarzyna Myrek <katarzyna(a)myrek.pl> CC: ceph-users(a)ceph.io Asunto: [ceph-users] Re: RGW and the orphans Hi Katarzyna, Incomplete multipart uploads are not considered orphans. With respect to the 404s…. Which version of ceph are you running? What tooling are you using to list and cancel? Can you provide a console transcript of the listing and cancelling? Thanks, Eric -- J. Eric Ivancich he / him / his Red Hat Storage Ann Arbor, Michigan, USA _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io