- ceph-users - lists.ceph.io

by Matt Benjamin

Hi Folks, I don't know of a downstream issue that looks like this, and we've upstreamed every fix for bucket listing and cleanup we have. We are pursuing a space leak believed to arise in "radosgw-admin bucket rm --purge-objects" but not a non-terminating listing. The only upstream release not planned to get a backport of orphans list tools is Luminous. I thought backport to Octopus was already done by the backport team? regards, Matt On Thu, Aug 6, 2020 at 2:40 PM EDH - Manuel Rios <mriosfer(a)easydatahost.com> wrote: > > You'r not the only one affected by this issue > > As far as i know several huge companies hitted this bug too, but private patches or tools are not public released. > > This is caused for the a resharding process during upload in previous versions. > > Workarround for us.: > > - Delete objects of the bucket at rados level. > - Delete the index file of the bucket. > > Pray to god to not happen again. > > Still pending backporting to Nautilus of the new experimental tool to find orphans in RGW > > Maybe @Matt Benjamin can give us and ETA for get ready that tool backported... > > Regards > > > > -----Mensaje original----- > De: Andrei Mikhailovsky <andrei(a)arhont.com> > Enviado el: jueves, 6 de agosto de 2020 13:55 > Para: ceph-users <ceph-users(a)ceph.io> > Asunto: [ceph-users] Re: RGW unable to delete a bucket > > BUMP... > > > ----- Original Message ----- > > From: "Andrei Mikhailovsky" <andrei(a)arhont.com> > > To: "ceph-users" <ceph-users(a)ceph.io> > > Sent: Tuesday, 4 August, 2020 17:16:28 > > Subject: [ceph-users] RGW unable to delete a bucket > > > Hi > > > > I am trying to delete a bucket using the following command: > > > > # radosgw-admin bucket rm --bucket=<bucket-name> --purge-objects > > > > However, in console I get the following messages. About 100+ of those > > messages per second. > > > > 2020-08-04T17:11:06.411+0100 7fe64cacf080 1 > > RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket > > listing requires read #1 > > > > > > The command has been running for about 35 days days and it still > > hasn't finished. The size of the bucket is under 1TB for sure. Probably around 500GB. > > > > I have recently removed about a dozen of old buckets without any > > issues. It's this particular bucket that is being very stubborn. > > > > Anything I can do to remove it, including it's objects and any orphans > > it might have? > > > > > > Thanks > > > > Andrei > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309

3 years, 9 months

2
1
0 0

DocuBetter Meeting this week -- 12 Aug 2020 0830 PDT

by John Zachary Dover

There is a general documentation meeting called the "DocuBetter Meeting", and it is held every two weeks. The next DocuBetter Meeting will be on 12 Aug 2020 at 0830 PDT, and will run for thirty minutes. Everyone with a documentation-related request or complaint is invited. The meeting will be held here: https://bluejeans.com/908675367 This meeting will cover the reorganization of the Ceph website as well as Zac's recently-developed workflow that aims to make it possible to move the good ideas from stale or poorly-formed documentation PRs into the documentation more quickly. Send documentation-related requests and complaints to me by replying to this email and CCing me at zac.dover(a)gmail.com. The next DocuBetter meeting is scheduled for: 12 Aug 2020 0830 PDT 12 Aug 2020 1630 UTC 13 Aug 2020 0230 AEST Etherpad: https://pad.ceph.com/p/Ceph_Documentation Zac's docs whiteboard: https://pad.ceph.com/p/docs_whiteboard Meeting: https://bluejeans.com/908675367

3 years, 9 months

1
0
0 0

RGW 14.2.10 Regresion? ordered bucket listing requires read #1

by EDH - Manuel Rios

Hi, We got our cluster updated to the lastest versión 14.2.10 Checking rgw logs after 14.2.10 upgrade 2020-08-10 10:21:49.186 7f74cd7db700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.188 7f75eca19700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.189 7f7479f34700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.193 7f74d77ef700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.196 7f753c0b8700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.197 7f74cd7db700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.197 7f745d6fb700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 Looks like this -> https://tracker.ceph.com/issues/42358 Regards Manuel

3 years, 9 months

1
0
0 0

What The Benefit Of Choosing Epson Customer Service?

by mary smith

In order get benefitted by choosing the correct and dependable Epson Customer Service service providers, it is strongly recommended to have a healthy research before getting involved with any of such service providers as there are several fake companies are also there claiming to be the best but actually not. https://www.epsonprintersupportpro.net/

3 years, 9 months

1
0
0 0

Get To Be Acquainted With HP Support Assistant In A Proper Manner.

by mary smith

In order to be acquainted with the HP Support Assistant in a proper manner, you should not leave any stone unturned in approaching the certified printer experts who will professionally assist you the exact troubleshooting solution to your problems in an effective manner. You can approach them by just using different convenient means at anytime. https://www.amiytech.com/hp-support-assistant/

3 years, 9 months

1
0
0 0

OSDs flapping since upgrade to 14.2.10

by Ingo Reimann

Hi list, since our upgrade 14.2.9 -> 14.2.10 we observe flapping OSDs: * The mons claim every few minutes: 2020-08-07 09:49:09.783648 osd.243 (osd.243) 246 : cluster [WRN] Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:04:40.753704 osd.243 (osd.243) 248 : cluster [WRN] Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:07:21.187945 osd.253 (osd.253) 469 : cluster [WRN] Monitor daemon marked osd.253 down, but it is still running 2020-08-07 10:04:35.440547 mon.cephmon01 (mon.0) 390132 : cluster [DBG] osd.243 reported failed by osd.33 2020-08-07 10:04:35.508412 mon.cephmon01 (mon.0) 390133 : cluster [DBG] osd.243 reported failed by osd.187 2020-08-07 10:04:35.508529 mon.cephmon01 (mon.0) 390134 : cluster [INF] osd.243 failed (root=default,datacenter=of,row=row-of-02,host=cephosd16) (2 reporters from different host after 44.000150 >= grace 25.935545) 2020-08-07 10:04:35.695171 mon.cephmon01 (mon.0) 390135 : cluster [DBG] osd.243 reported failed by osd.203 2020-08-07 10:04:35.771704 mon.cephmon01 (mon.0) 390136 : cluster [DBG] osd.243 reported failed by osd.163 2020-08-07 10:04:41.588530 mon.cephmon01 (mon.0) 390148 : cluster [INF] osd.243 [v2:10.198.10.16:6882/6611,v1:10.198.10.16:6885/6611] boot 2020-08-07 10:04:40.753704 osd.243 (osd.243) 248 : cluster [WRN] Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:04:40.753712 osd.243 (osd.243) 249 : cluster [DBG] map e2683535 wrongly marked me down at e2683534 osd.33 says: 2020-08-07 10:04:35.437 7fcaaa4f3700 -1 osd.33 2683533 heartbeat_check: no reply from 10.198.10.16:6802 osd.243 since back 2020-08-07 10:03:51.223911 front 2020-08-07 10:03:51.224322 (oldest deadline 2020-08-07 10:04:35.322704) osd.243 says: 2020-08-07 10:03:55.065 7f0d33911700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 2020-08-07 10:03:55.065 7f0d34112700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 [.. ~3000(!) Lines ..] 2020-08-07 10:04:33.644 7f0d33110700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 2020-08-07 10:04:33.688 7f0d13acb700 0 bluestore(/var/lib/ceph/osd/ceph-243) log_latency_fn slow operation observed for upper_bound, latency = 20.9013s, after = omap_iterator(cid = 19.58a_head, oid = #19:51a21a27::: .dir.default.223091333.1.3:head#) 2020-08-07 10:04:33.688 7f0d13acb700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7f0d13acb700' had timed out after 15 2020-08-07 10:04:40.748 7f0d2279b700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.243 down, but it is still running 2020-08-07 10:04:40.748 7f0d2279b700 0 log_channel(cluster) log [DBG] : map e2683535 wrongly marked me down at e2683534 * as a consequence, old deep-scrubs did not finish, because they would be interrupted -> ' pgs not deep-scrubbed in time' for the latter, I increased the op-thread-timeout back to the pre 12(!).2.11 value of 30 i`m am not sure, if we really have a problem, but it does not look healthy. Any ideas, thoughts? regards, Ingo -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support(a)dunkel.de https://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001

3 years, 9 months

4
6
0 0

i would love more this content

by chiaranjun＠gmail.com

Nice to be visiting your blog again, it has been months for me. Well this article that i've been waited for so long. I need this article to complete my assignment in the college, and it has same topic with your article. Thanks, great share.Here are some ineteresting content i have to share as well; <a href="https://pomeranianhaustierhaus.com/product/pomeranian-for-sale-near-me/"rel="dofollow">pomeranian for sale near me</a> <a href="https://pomeranianhaustierhaus.com/product/pomeranian-chihuahua-mix/"rel="dofollow">pomeranian chihuahua mix</a> <a href="https://pomeranianhaustierhaus.com/product/pomeranian-for-sale/"rel="dofollow">pomeranian for sale</a> <a href="https://pomeranianhaustierhaus.com/product/pomeranian-on-sale/"rel="dofollow">pomeranian on sale</a> <a href="https://pomeranianhaustierhaus.com/product/pomeranian-black-and-white/"rel="dofollow">pomeranian black and white</a> <a href="https://pomeranianhaustierhaus.com/product/pomeranian-miniature/"rel="dofollow">pomeranian miniature</a> <a href="https://pomeranianhaustierhaus.com/product/pomeranian-black/"rel="dofollow">pomeranian black</a> <a href="https://pomeranianhaustierhaus.com/product/pomeranian-adoption/"rel="dofollow">pomeranian adoption</a> <a href="https://pomeranianhaustierhaus.com/pomeranian-puppy/"rel="dofollow">pomeranian puppy</a>

3 years, 9 months

4
5
0 0

Block Network IP Of Wireless HP Printer Using HP Support Assistant.

by mary smith

Yes, you should! If you are one of those who are have a word with the professionals who are capable of resolving the whole host of network problems. You can make use of HP Support Assistant who will solve the whole host of wireless printer problems. https://www.amiytech.com/hp-support-assistant/

3 years, 9 months

1
0
0 0

Can I Unlock Yahoo Account without a telephone number?

by mary smith

While you are making a record on Yahoo messaging stage, at that point it will request an elective email address just as a portable number to send a confirmation code. Along these lines, you have compulsory to give all subtleties accurately and furthermore set a secret key. In the event that you feel that, would I be able to Unlock Yahoo Account without a telephone number? Thus, indeed, you can, yet you need to choose an elective alternative to get a confirmation code. https://www.contact-customer-service.net/blog/unlock-yahoo-account/

3 years, 9 months

1
0
0 0

How to Get Refund for Cash App Payment Pending

by itisghana＠gmail.com

Welcome to Cash App Support Center +1-888-526-0829 Welcome to Cash App Support Center +1-888-526-0829 Welcome to Cash App Support Center +1-888-526-0829 https://cashapphelp.support/

3 years, 9 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users