Re: ceph-users Digest, Vol 81, Issue 39 Re:RadosGW cant list objects when there are too many of them

17 Oct 2019

      Hi Arash,
  If the number of objects in a bucket are too large in the order of
millions, a paginated listing approach works better.
There are also ceratin RGW configs, that controls on how big a RGW response
(in terms of number objects can be, by default I believe this is 1000)
The code for Paginated Listing (Snippet can be modified):-

*"*

*try:         buckethandle = s3_conn_src.get_bucket(bucket_name)*
*         while True:*

*             keys = buckethandle.get_all_keys(max_keys=1000,marker =
marker)             for k in keys:*
*                     #do operation on keys (which are the objects)*
*                     print k.name <http://k.name>*
*                    #update marker*
*                    marker = k.name <http://k.name>*

*             if keys.is_truncated is False:                 print
"Breaking"                 break*
*except Exception, e:*
*          print e*
*"*

Thanks
Romit Misra

Thanks
Romit

On Thu, Oct 17, 2019 at 4:18 PM <ceph-users-request@ceph.io> wrote:
...
Send ceph-users mailing list submissions to
        ceph-users@ceph.io
To subscribe or unsubscribe via email, send a message with subject or
body 'help' to
        ceph-users-request@ceph.io
You can reach the person managing the list at
        ceph-users-owner@ceph.io
When replying, please edit your Subject line so it is more specific
than "Re: Contents of ceph-users digest..."
Today's Topics:
1. RadosGW cant list objects when there are too many of them
      (Arash Shams)
   2. Re: Recovering from a Failed Disk (replication 1) (Burkhard Linke)
   3. Re: RGW blocking on large objects (Paul Emmerich)
   4. Re: RadosGW cant list objects when there are too many of them
      (Paul Emmerich)
   5. Re: Recovering from a Failed Disk (replication 1) (Frank Schilder)
----------------------------------------------------------------------
Date: Thu, 17 Oct 2019 07:19:12 +0000
From: Arash Shams <ara4sh@hotmail.com>
Subject: [ceph-users] RadosGW cant list objects when there are too
        many of them
To: "ceph-users@ceph.io" <ceph-users@ceph.io>
Message-ID:  <LNXP265MB0508FF1F47CB5EA9C29219FA926D0@LNXP265MB0508.GBR
        P265.PROD.OUTLOOK.COM>
Content-Type: multipart/alternative;    boundary="_000_LNXP265MB0508FF
        1F47CB5EA9C29219FA926D0LNXP265MB0508GBRP_"
--_000_LNXP265MB0508FF1F47CB5EA9C29219FA926D0LNXP265MB0508GBRP_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Dear All
I have a bucket with 5 million Objects and I cant list objects with
radosgw-admin bucket list --bucket=3Dbucket | jq .[].name
or listing files using boto3
s3 =3D boto3.client('s3',
                      endpoint_url=3Dcredentials['endpoint_url'],
                      aws_access_key_id=3Dcredentials['access_key'],
                      aws_secret_access_key=3Dcredentials['secret_key'])
response =3D s3.list_objects_v2(Bucket=3Dbucket_name)
    for item in response['Contents']:
        print(item['Key'])
what is the solution ? how can I find list of my objects ?
--_000_LNXP265MB0508FF1F47CB5EA9C29219FA926D0LNXP265MB0508GBRP_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html;
charset=3Diso-8859-=
1">
<style type=3D"text/css" style=3D"display:none;"> P
{margin-top:0;margin-bo=
ttom:0;} </style>
</head>
<body dir=3D"ltr">
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size:
12pt;=
 color: rgb(0, 0, 0);">
Dear All <br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size:
12pt;=
 color: rgb(0, 0, 0);">
<br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size:
12pt;=
 color: rgb(0, 0, 0);">
I have a bucket with 5 million Objects and I cant list objects with <br>
radosgw-admin bucket list --bucket=3Dbucket | jq .[].name<br>
or listing files using boto3 <br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size:
12pt;=
 color: rgb(0, 0, 0);">
<span><br>
</span></div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size:
12pt;=
 color: rgb(0, 0, 0);">
<span>    s3 =3D boto3.client('s3',<br>
</span>
<div>                   
=
  endpoint_url=3Dcredentials['endpoint_url'],<br>
</div>
<div>                   
=
  aws_access_key_id=3Dcredentials['access_key'],<br>
</div>
<div>                   
=
  aws_secret_access_key=3Dcredentials['secret_key'])<br>
</div>
<div><br>
</div>
<div>    response =3D
s3.list_objects_v2(Bucket=3Dbucket_name)<br=
...
</div>
<div>    for item in response['Contents']:<br>
</div>
<span>        print(item['Key'])</span><br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size:
12pt;=
 color: rgb(0, 0, 0);">
<br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size:
12pt;=
 color: rgb(0, 0, 0);">
what is the solution ? how can I find list of my objects ?</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size:
12pt;=
 color: rgb(0, 0, 0);">
<br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size:
12pt;=
 color: rgb(0, 0, 0);">
<br>
</div>
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size:
12pt;=
 color: rgb(0, 0, 0);">
<br>
</div>
</body>
</html>
--_000_LNXP265MB0508FF1F47CB5EA9C29219FA926D0LNXP265MB0508GBRP_--
------------------------------
Date: Thu, 17 Oct 2019 10:18:11 +0200
From: Burkhard Linke <Burkhard.Linke@computational.bio.uni-giessen.de>
Subject: [ceph-users] Re: Recovering from a Failed Disk (replication
        1)
To: ceph-users@ceph.io
Message-ID: <f341533b-cc7a-865d-0440-79084e0c5707@computational.bio.un
        i-giessen.de>
Content-Type: multipart/alternative;
        boundary="------------71A0D501B0D56489A2F673CA"
This is a multi-part message in MIME format.
--------------71A0D501B0D56489A2F673CA
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Hi,
On 10/17/19 5:56 AM, Ashley Merrick wrote:
...
I think your better off doing the DD method, you can export and import
a PG at a time (ceph-objectstore-tool)
But if the disk is failing a DD is probably your best method.
In case of hardware problems or broken sectors, I would recommend
'dd_rescue' instead of dd. It can handle broken sectors, automatic
retries, skipping etc.
You will definitely need a second disk to rescue to.
Regards,
Burkhard
--------------71A0D501B0D56489A2F673CA
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p>Hi,</p>
    <p><br>
    </p>
    <div class="moz-cite-prefix">On 10/17/19 5:56 AM, Ashley Merrick
      wrote:<br>
    </div>
    <blockquote type="cite"
      cite="
mid:16dd7dc0f88.e12acd11469078.7469588663842930162@amerrick.co.uk">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      <div style="font-family: Verdana, Arial, Helvetica, sans-serif;
        font-size: 10pt;">I think your better off doing the DD method,
        you can export and import a PG at a time
(ceph-objectstore-tool)<br>
        <br>
        But if the disk is failing a DD is probably your best method.<br>
      </div>
    </blockquote>
    <p><br>
    </p>
    <p>In case of hardware problems or broken sectors, I would recommend
      'dd_rescue' instead of dd. It can handle broken sectors, automatic
      retries, skipping etc.</p>
    <p><br>
    </p>
    <p>You will definitely need a second disk to rescue to.</p>
    <p><br>
    </p>
    <p>Regards,</p>
    <p>Burkhard</p>
    <br>
  </body>
</html>
--------------71A0D501B0D56489A2F673CA--
------------------------------
Date: Thu, 17 Oct 2019 11:50:37 +0200
From: Paul Emmerich <paul.emmerich@croit.io>
Subject: [ceph-users] Re: RGW blocking on large objects
To: Robert LeBlanc <robert@leblancnet.us>
Cc: ceph-users <ceph-users@ceph.io>
Message-ID:
        <
CAD9yTbFSpxo1cMAAZ56YcwKxY2dcv0KRPcWYapT2fLmYhmrLkg@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
On Thu, Oct 17, 2019 at 12:17 AM Robert LeBlanc <robert@leblancnet.us>
wrote:
...
On Wed, Oct 16, 2019 at 2:50 PM Paul Emmerich <paul.emmerich@croit.io>
...
...
On Wed, Oct 16, 2019 at 11:23 PM Robert LeBlanc <robert@leblancnet.us>
wrote:
...
...
On Tue, Oct 15, 2019 at 8:05 AM Robert LeBlanc <robert@leblancnet.us>
wrote:
...
...
On Mon, Oct 14, 2019 at 2:58 PM Paul Emmerich <
wrote:
paul.emmerich@croit.io> wrote:
...
...
...
...
...
Could the 4 GB GET limit saturate the connection from rgw to
Ceph?
...
Simple to test: just rate-limit the health check GET
I don't think so, we have dual 25Gbp in a LAG, so Ceph to RGW has
multiple paths, but we aren't balancing on port yet, so RGW to
HAProxy
is probably limited to one link.
...
Did you increase "objecter inflight ops" and "objecter inflight
op bytes"?
You absolutely should adjust these settings for large RGW setups,
defaults of 1024 and 100 MB are way too low for many RGW setups,
we
default to 8192 and 800MB
On Nautilus the defaults already seem to be:
objecter_inflight_op_bytes                                 104857600
                      default
= 100MiB
...
objecter_inflight_ops                                      24576
                      default
not sure where you got this from, but the default is still 1024 even
in master:
https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/s...
Looks like it is overridden in
https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/s...
you are right, this is new in Nautilus. Last time I had to play around
with these settings was indeed on a Mimic deployment.
...
I'm just not
understanding how your suggestions would help, the problem doesn't
seem to be on the RADOS side (which it appears your tweaks target),
but on the HTTP side as an HTTP health check takes a long time to come
back when a big transfer is going on.
I was guessing a bottleneck on the RADOS side because you mentioned
that you tried both civetweb and beast, somewhat unlikely to run into
the exact same problem with both
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
------------------------------
Date: Thu, 17 Oct 2019 12:00:20 +0200
From: Paul Emmerich <paul.emmerich@croit.io>
Subject: [ceph-users] Re: RadosGW cant list objects when there are too
        many of them
To: Arash Shams <ara4sh@hotmail.com>
Cc: "ceph-users@ceph.io" <ceph-users@ceph.io>
Message-ID:
        <CAD9yTbGoCGPh=
Ba5tQqBvCLu9uUKo0KmjR9E3ayo6DkR-E2bxQ@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Listing large buckets is slow due to S3 ordering requirements, it's
approximately O(n^2).
However, I wouldn't consider 5M to be a large bucket, it should go to
only ~50 shards which should still perform reasonable. How fast are
your metadata OSDs?
Try --allow-unordered in radosgw-admin to get an unordered result
which is only O(n) as you'd expect.
For boto3: I'm not sure if v2 object listing is available yet (I think
it has only been merged into master but has not yet made it into a
release?). It doesn't support unordered listing but there has been
some work to implement it there, not sure about the current state.
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Thu, Oct 17, 2019 at 9:19 AM Arash Shams <ara4sh@hotmail.com> wrote:
...
Dear All
I have a bucket with 5 million Objects and I cant list objects with
radosgw-admin bucket list --bucket=bucket | jq .[].name
or listing files using boto3
s3 = boto3.client('s3',
                      endpoint_url=credentials['endpoint_url'],
                      aws_access_key_id=credentials['access_key'],
                      aws_secret_access_key=credentials['secret_key'])
response = s3.list_objects_v2(Bucket=bucket_name)
    for item in response['Contents']:
        print(item['Key'])
what is the solution ? how can I find list of my objects ?
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
------------------------------
Date: Thu, 17 Oct 2019 10:46:53 +0000
From: Frank Schilder <frans@dtu.dk>
Subject: [ceph-users] Re: Recovering from a Failed Disk (replication
        1)
To: vladimir franciz blando <vladimir.blando@gmail.com>,
        "ceph-users@ceph.io"   <ceph-users@ceph.io>
Message-ID: <58e22bc6345b48718dbce7238a06e35d@dtu.dk>
Content-Type: text/plain; charset="utf-8"
You probably need to attempt a physical data rescue. Data access will be
lost until done.
First thing is shut down the OSD to avoid any further damage to the disk.
Second thing is to try ddrescue, repair data on a copy if possible and
then create a clone on a new disk from the copy.
If this doesn't help and you really need that last bit of data, you might
need support from one of those companies that restore disk data with
electron microscopy.
I successfully transferred OSDs between disks using ddrescue.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: vladimir franciz blando <vladimir.blando@gmail.com>
Sent: 17 October 2019 05:29:13
To: ceph-users@ceph.io
Subject: [ceph-users] Recovering from a Failed Disk (replication 1)
Hi,
I have a not ideal setup on one of my cluster,  3 ceph  nodes but using
replication 1 on all pools (don't ask me why replication 1, it's a long
story).
So it has come to this situation that a disk keeps on crashing, possible a
hardware failure and I need to recover from that.
What's my best option for me to recover the data from the failed disk and
transfer it to the other healthy disks?
This cluster is using Firefly
- Vlad
[
https://mailfoogae.appspot.com/t?sender=admxhZGltaXIuYmxhbmRvQGdtYWlsLmNvbQ%3D%3D&type=zerocontent&guid=976ce724-3894-4a75-b591-dca017bdf19e]ᐧ
<https://mailfoogae.appspot.com/t?sender=admxhZGltaXIuYmxhbmRvQGdtYWlsLmNvbQ%3D%3D&type=zerocontent&guid=976ce724-3894-4a75-b591-dca017bdf19e]%E1%90%A7>
------------------------------
Subject: Digest Footer
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
------------------------------
End of ceph-users Digest, Vol 81, Issue 39
******************************************
-- 

*-----------------------------------------------------------------------------------------*

*This email and any files transmitted with it are confidential and 
intended solely for the use of the individual or entity to whom they are 
addressed. If you have received this email in error, please notify the 
system manager. This message contains confidential information and is 
intended only for the individual named. If you are not the named addressee, 
you should not disseminate, distribute or copy this email. Please notify 
the sender immediately by email if you have received this email by mistake 
and delete this email from your system. If you are not the intended 
recipient, you are notified that disclosing, copying, distributing or 
taking any action in reliance on the contents of this information is 
strictly prohibited.*****

 ****

*Any views or opinions presented in this 
email are solely those of the author and do not necessarily represent those 
of the organization. Any information on shares, debentures or similar 
instruments, recommended product pricing, valuations and the like are for 
information purposes only. It is not meant to be an instruction or 
recommendation, as the case may be, to buy or to sell securities, products, 
services nor an offer to buy or sell securities, products or services 
unless specifically stated to be so on behalf of the Flipkart group. 
Employees of the Flipkart group of companies are expressly required not to 
make defamatory statements and not to infringe or authorise any 
infringement of copyright or any other legal right by email communications. 
Any such communication is contrary to organizational policy and outside the 
scope of the employment of the individual concerned. The organization will 
not accept any liability in respect of such communication, and the employee 
responsible will be personally liable for any damages or other liability 
arising.*****

 ****

*Our organization accepts no liability for the 
content of this email, or for the consequences of any actions taken on the 
basis of the information *provided,* unless that information is 
subsequently confirmed in writing. If you are not the intended recipient, 
you are notified that disclosing, copying, distributing or taking any 
action in reliance on the contents of this information is strictly 
prohibited.*

_-----------------------------------------------------------------------------------------_

Romit Misra

tags

participants (1)