- ceph-users - lists.ceph.io

Size and capacity calculations questions
by Jochen Schulz 11 Dec '19

11 Dec '19

Hi! We have a ceph cluster with 42 OSD in production as a server providing mainly home-directories of users. Ceph is 14.2.4 nautilus. We have 3 pools. One images (for rbd images) a cephfs_metadata and a cephfs_data pool. Our raw data is about 5.6T. All pools have replica size 3 and there are only very little snapshots in the rbd images pool, the cephfspool doesnt use snapshots. How is it possible that the status tells us, that 21T/46T is used, because thats much more than 3 times the raw size. Also, to make that more confusing, there as at least half of the cluster free, and we get pg backfill_toofull after we added some OSDs lately. The ceph dashboard tells aus the pool ist 82 % full and has only 4.5 T free. The autoscale module seems to calculate the 20T times 3 for the space needed and thus has wrong numbers (see below). Status of the cluster is added below too. how can these size/capacity numbers be explained? and, would be there a recommendation to change something? Thank you in advance! best Jochen # ceph -s cluster: id: 2b16167f-3f33-4580-a0e9-7a71978f403d health: HEALTH_ERR Degraded data redundancy (low space): 1 pg backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 2 pools have too many placement groups services: mon: 4 daemons, quorum jade,assam,matcha,jasmine (age 2d) mgr: earl(active, since 24h), standbys: assam mds: cephfs:1 {0=assam=up:active} 1 up:standby osd: 42 osds: 42 up (since 106m), 42 in (since 115m); 30 remapped pgs data: pools: 3 pools, 2048 pgs objects: 29.80M objects, 5.6 TiB usage: 21 TiB used, 25 TiB / 46 TiB avail pgs: 1164396/89411013 objects misplaced (1.302%) 2018 active+clean 22 active+remapped+backfill_wait 7 active+remapped+backfilling 1 active+remapped+backfill_wait+backfill_toofull io: client: 1.7 KiB/s rd, 516 KiB/s wr, 0 op/s rd, 28 op/s wr recovery: 9.2 MiB/s, 41 objects/s # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE images 354.2G 3.0 46100G 0.0231 1.0 1024 32 warn cephfs_metadata 13260M 3.0 595.7G 0.0652 1.0 512 8 warn cephfs_data 20802G 3.0 46100G 1.3537 1.0 512 warn

5 9

getfattr problem on ceph-fs
by Frank Schilder 11 Dec '19

11 Dec '19

I have a strange problem with ceph fs and extended attributes. I have two Centos machines where I mount cephfs in exactly the same way (I manually executed the exact same mount command on both machines). On one of the machines, getfattr returns this: [root@ceph-01 ~]# getfattr -d -m 'ceph.*' /mnt/cephfs/hpc/home getfattr: Removing leading '/' from absolute path names # file: mnt/cephfs/hpc/home ceph.dir.entries="49" ceph.dir.files="1" ceph.dir.rbytes="77816237666910" ceph.dir.rctime="1575978038.0976848840" ceph.dir.rentries="6673312" ceph.dir.rfiles="6271408" ceph.dir.rsubdirs="401904" ceph.dir.subdirs="48" and on the other I get nothing: [root@gnosis ~]# getfattr -d -m 'ceph.*' /mnt/cephfs/hpc/home No error message, just nothing. The only difference is, that ceph-01 was kickstarted with Centos7.6 while gnosis was kickstarted with Centos7.7. Otherwise, both machines are deployed identically. getfattr is the same version on both. Kernel versions are ceph-01:5.0.2-1.el7.elrepo.x86_64 and gnosis:5.4.2-1.el7.elrepo.x86_64. Does anyone have a pointer what to look for? Thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 4

Shouldn't Ceph's documentation be "per version"?
by Rodrigo Severo - Fábrica 10 Dec '19

10 Dec '19

Hi, Shouldn't Ceph's documentation be presented "per version"? I believe there might be documentation for Ceph per version but I can't see in Ceph documentation site how to easily see each version's docs. Regards, Rodrigo Severo

2 1

Cephalocon 2020
by Sage Weil 10 Dec '19

10 Dec '19

Hi everyone, The next Cephalocon is coming up on March 3-5 in Seoul! The CFP is open until Friday (get your talks in!). We expect to have the program ready for the first week of January. Registration (early bird) will be available soon. We're also looking for sponsors for the conference. The prospectus is available here: https://ceph.io/wp-content/uploads/2019/12/sponsor-Cephalocon20-112719.pdf Thanks! sage

1 1

[object gateway] setting storage class does not move object to correct backing pool?
by Gerdriaan Mulder 10 Dec '19

10 Dec '19

Hi, If I change the storage class of an object via s3cmd, the object's storage class is reported as being changed. However, when inspecting where the objects are placed (via `rados -p <pool> ls`, see further on), the object seems to be retained in the original pool. The idea behind this test setup is to simulate two storage locations, one based on SSDs or similar flash storage, the other on slow HDDs. We want to be able to alter the storage location of objects on the fly, typically only from fast to slow storage. The object should then only reside on slow storage. The setup is as follows on Nautilus (Ubuntu 16.04, see <https://gist.github.com/mrngm/bba6ffdc545bfa52ebf79d6d2c002a6d> for the full dump): <<<<<<<< root@node1:~# ceph -s health: HEALTH_OK mon: 3 daemons, quorum node1,node3,node5 (age 12d) mgr: node2(active, since 6d), standbys: node4 osd: 4 osds: 4 up (since 12d), 4 in (since 12d) rgw: 1 daemon active (node1) pools: 7 pools, 296 pgs objects: 229 objects, 192 KiB usage: 3.2 GiB used, 6.8 GiB / 10 GiB avail pgs: 296 active+clean root@node1:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.00970 root default -16 0.00970 datacenter nijmegen -3 0.00388 host node2 0 hdd 0.00388 osd.0 up 1.00000 1.00000 -5 0.00388 host node3 1 hdd 0.00388 osd.1 up 1.00000 1.00000 -7 0.00098 host node4 2 ssd 0.00098 osd.2 up 1.00000 1.00000 -9 0.00098 host node5 3 ssd 0.00098 osd.3 up 1.00000 1.00000 root@node1:~# ceph osd pool ls detail pool 1 'tier1-ssd' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 [snip] application rgw pool 2 'tier2-hdd' replicated size 1 min_size 1 crush_rule 2 object_hash rjenkins pg_num 128 pgp_num 128 [snip] application rgw pool 3 '.rgw.root' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 [snip] application rgw pool 4 'default.rgw.control' replicated size 2 min_size 1 crush_rule 0 [snip] application rgw pool 5 'default.rgw.meta' replicated size 2 min_size 1 crush_rule 0 [snip] application rgw pool 6 'default.rgw.log' replicated size 2 min_size 1 crush_rule 0 [snip] application rgw pool 7 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 [snip] application rgw root@node1:~# ceph osd pool application get # compacted tier1-ssd => rgw {} tier2-hdd => rgw {} .rgw.root => rgw {} default.rgw.control => rgw {} default.rgw.meta => rgw {} default.rgw.log => rgw {} default.rgw.buckets.index => rgw {} root@node1:~# radosgw-admin zonegroup placement list [ { "key": "default-placement", "val": { "name": "default-placement", "tags": [], "storage_classes": [ "SPINNING_RUST", "STANDARD" ] } } ] root@node1:~# radosgw-admin zone placement list [ { "key": "default-placement", "val": { "index_pool": "default.rgw.buckets.index", "storage_classes": { "SPINNING_RUST": { "data_pool": "tier2-hdd" }, "STANDARD": { "data_pool": "tier1-ssd" } }, "data_extra_pool": "default.rgw.buckets.non-ec", "index_type": 0 } } ] ======== I can also post the relevant s3cmd commands for putting objects and setting the storage class, but perhaps this is already enough information. Please let me know. <<<<<<<< root@node1:~# rados -p tier1-ssd ls ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1_darthvader.png ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1_2019-10-15-090436_1254x522_scrubbed.png ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1_kanariepiet.jpg root@node1:~# rados -p tier2-hdd ls ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1__shadow_.FEruUOZaVJXJcOG-e2tO1xcInNzoEvN_0 $ s3cmd info s3://bucket/kanariepiet.jpg [snip] Last mod: Tue, 10 Dec 2019 08:09:58 GMT Storage: STANDARD [snip] $ s3cmd info s3://bucket/darthvader.png [snip] Last mod: Wed, 04 Dec 2019 10:35:14 GMT Storage: SPINNING_RUST [snip] $ s3cmd info s3://bucket/2019-10-15-090436_1254x522_scrubbed.png [snip] Last mod: Tue, 10 Dec 2019 10:33:24 GMT Storage: STANDARD [snip] ========== Any thoughts on what might occur here? Best regards, Gerdriaan Mulder

3 4

nautilus radosgw fails with pre jewel buckets - index objects not at right place
by Ingo Reimann 10 Dec '19

10 Dec '19

Hi, 2 years after my issue [ https://tracker.ceph.com/issues/22928 | https://tracker.ceph.com/issues/22928 ] the next one fires back. The Problem: Old Buckets have their index and data in rgw.buckets: root@cephrgw01:~# radosgw-admin metadata get bucket:testtesttesty { "key": "bucket:testtesttesty", "ver": { "tag": "_E_OHNhD28Zu1DeuvyGq8Q8b", "ver": 1 }, "mtime": "2013-11-11 09:25:56.000000Z", "data": { "bucket": { "name": "testtesttesty", "marker": "default.2542971.19", "bucket_id": "default.2542971.19", "tenant": "", "explicit_placement": { "data_pool": "rgw.buckets", "data_extra_pool": "", "index_pool": "rgw.buckets" } }, "owner": "123", "creation_time": "2013-11-11 09:25:56.000000Z", "linked": "true", "has_bucket_info": "false" } } After upgrade from luminous to nautilus i get 400(InvalidArgument) and "NOTICE: invalid dest placement" in the radosgw-log on access to the buckets My zone defines: root@cephrgw01:~# radosgw-admin zone get { "id": "default", "name": "default", "domain_root": ".rgw", "control_pool": ".rgw.control", "gc_pool": ".rgw.gc", "lc_pool": ".log:lc", "log_pool": ".log", "intent_log_pool": ".intent-log", "usage_log_pool": ".usage", "reshard_pool": ".log:reshard", "user_keys_pool": ".users", "user_email_pool": ".users.email", "user_swift_pool": ".users.swift", "user_uid_pool": ".users.uid", "otp_pool": "default.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "rgw.buckets" } }, "data_extra_pool": "rgw.buckets.non-ec", "index_type": 0 } } ], "metadata_heap": ".rgw.meta", "realm_id": "*********************c" } Now i am a little bit lost. I added a new placement to my zone and zonegroup radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id pre-jewel radosgw-admin zone placement add --rgw-zonegroup default --placement-id pre-jewel --data-pool rgw.buckets --index-pool rgw.buckets data-extra-pool "" radosgw-admin period update --commit root@cephrgw01:~# radosgw-admin zone get { "id": "default", "name": "default", "domain_root": ".rgw", "control_pool": ".rgw.control", "gc_pool": ".rgw.gc", "lc_pool": ".log:lc", "log_pool": ".log", "intent_log_pool": ".intent-log", "usage_log_pool": ".usage", "reshard_pool": ".log:reshard", "user_keys_pool": ".users", "user_email_pool": ".users.email", "user_swift_pool": ".users.swift", "user_uid_pool": ".users.uid", "otp_pool": "default.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "rgw.buckets" } }, "data_extra_pool": "rgw.buckets.non-ec", "index_type": 0 } }, { "key": "pre-jewel", "val": { "index_pool": "rgw.buckets", "storage_classes": { "STANDARD": { "data_pool": "rgw.buckets" } }, "data_extra_pool": "", "index_type": 0 } } ], "metadata_heap": ".rgw.meta", "realm_id": "****************c" } Nevertheless, only the luminous gateways may list my old buckets. As far as I can see, I may only change the placement_rule for new buckets. Is there any chance to make radosgw find the old indices and complete the upgrade to nautilus? Many thanks, Ingo -- Ingo Reimann [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support(a)dunkel.de http://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001

2 5

Re: osdmaps not trimmed until ceph-mon's
by Romit Misra 10 Dec '19

10 Dec '19

Hi Bryan and Dan, Had some similar observations, and wanted some data points from you as well if possible. 1. When you say down osd , is it down and in or down and out. 2. I see the osd accumulating map ranges, when continuous recovery is going on, due to osd flaps( that is some portion of the total of pgs in cluster are not in active+clean) state. 3.Another observation is that, although it may be one of the pools that see a osd churn, the map ranges seem to increase across all the OSd across all the pools. 4.I use the asok socket to dump the status, on osd Daemon which shows the map ranges. 5.Even I tried restarting Mon, one after another, and that seemed to reduce the space, but in a sporadic fashion. 6. Once there was some breathing room, in terms of OSd not flapping, and all PG being in active + clean state, the size trimmed down to almost optimal spaces. 7. I found some pointers at :- https://docs.ceph.com/docs/master/dev/mon-osdmap-prune/ 8. Th behaviour I am specifying us in 12.X Luminous release. 9. Would you care to share what commands you use to dump maps usage info from space perspective. Thanks Romit On Mon, 9 Dec 2019, 22:56 , <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. RGW listing millions of objects takes too much time (Arash Shams) > 2. Re: RGW listing millions of objects takes too much time > (Robert LeBlanc) > 3. ceph mgr daemon multiple ip addresses (Frank R) > 4. Re: osdmaps not trimmed until ceph-mon's restarted (if cluster has a > down osd) > (Bryan Stillwell) > > > ---------------------------------------------------------------------- > > Date: Mon, 9 Dec 2019 15:46:04 +0000 > From: Arash Shams <ara4sh(a)hotmail.com> > Subject: [ceph-users] RGW listing millions of objects takes too much > time > To: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <CWXP265MB128679DE9E6B2A6CA289B16492580(a)CWXP265MB1286.GBR > P265.PROD.OUTLOOK.COM> > Content-Type: multipart/alternative; boundary="_000_CWXP265MB128679 > DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_" > > --_000_CWXP265MB128679DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_ > Content-Type: text/plain; charset="iso-8859-1" > Content-Transfer-Encoding: quoted-printable > > Dear All, > > I have almost 30 million objects and I want to list them and index them > som= > ewhere else, > Im using boto3 with continuation Marker but it takes almost 9 hours > > can I run it in multiple threads to make it faster? what solution do you > su= > ggest to speedup this process, > > > Thanks > > > --_000_CWXP265MB128679DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_ > Content-Type: text/html; charset="iso-8859-1" > Content-Transfer-Encoding: quoted-printable > > <html> > <head> > <meta http-equiv=3D"Content-Type" content=3D"text/html; > charset=3Diso-8859-= > 1"> > <style type=3D"text/css" style=3D"display:none;"> P > {margin-top:0;margin-bo= > ttom:0;} </style> > </head> > <body dir=3D"ltr"> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > Dear All, </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > I have almost 30 million objects and I want to list them and index them > som= > ewhere else, </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > Im using boto3 with continuation Marker but it takes almost 9 hours</div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > can I run it in multiple threads to make it faster? what solution do you > su= > ggest to speedup this process, </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > Thanks </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > </body> > </html> > > --_000_CWXP265MB128679DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_-- > > ------------------------------ > > Date: Mon, 9 Dec 2019 08:23:55 -0800 > From: Robert LeBlanc <robert(a)leblancnet.us> > Subject: [ceph-users] Re: RGW listing millions of objects takes too > much time > To: Arash Shams <ara4sh(a)hotmail.com> > Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: > <CAANLjFoZu8LE0eH2vB9QArAOZJf4Ofm8V= > jAW-+ZPOFS-f9Oag(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="0000000000000e664e059947d3b2" > > --0000000000000e664e059947d3b2 > Content-Type: text/plain; charset="UTF-8" > > On Mon, Dec 9, 2019 at 7:47 AM Arash Shams <ara4sh(a)hotmail.com> wrote: > > > Dear All, > > > > I have almost 30 million objects and I want to list them and index them > > somewhere else, > > Im using boto3 with continuation Marker but it takes almost 9 hours > > > > can I run it in multiple threads to make it faster? what solution do you > > suggest to speedup this process, > > > > > > Thanks > > > > I've thought about indexing objects elsewhere as well. One thought I had > was hooking into the HTTP flow where a PUT or DEL would update the objects > in some kind of database (async of course). We could also gather stats with > GET and POST. Initially, my thoughts were to hook into haproxy since we > already use it, but possibly RGW if that is an option. That way it would > always be up to date and not have to do big scans on the buckets (our > buckets would not perform well with this). I haven't actually gotten to the > implementation phase of this idea. > > ---------------- > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > > --0000000000000e664e059947d3b2 > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"ltr"><div dir=3D"ltr">On Mon, Dec 9, 2019 at 7:47 AM Arash > Sham= > s <<a href=3D"mailto:ara4sh@hotmail.com">ara4sh(a)hotmail.com</a>> > wrot= > e:<br></div><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" > st= > yle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid > rgb(204,204,204);padd= > ing-left:1ex"> > > > > > <div dir=3D"ltr"> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > Dear All,=C2=A0</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > I have almost 30 million objects and I want to list them and index them > som= > ewhere else,=C2=A0</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > Im using boto3 with continuation Marker but it takes almost 9 hours</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > can I run it in multiple threads to make it faster? what solution do you > su= > ggest to speedup this process,=C2=A0</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > Thanks=C2=A0</div></div></blockquote><div > class=3D"gmail_quote"><br></div><= > div class=3D"gmail_quote">I've thought about indexing objects > elsewhere= > as well. One thought I had was hooking into the HTTP flow where a PUT or > D= > EL would update the objects in some kind of database (async of course). We > = > could also gather stats with GET and POST. Initially, my thoughts were to > h= > ook into haproxy since we already use it, but possibly RGW if that is an > op= > tion. That way it would always be up to date and not have to do big scans > o= > n the buckets (our buckets would not perform well with this). I > haven't= > actually gotten to the implementation phase of this idea.</div><br clear= > =3D"all"><div><div dir=3D"ltr" > class=3D"gmail_signature">----------------<b= > r>Robert LeBlanc<br></div></div><div>PGP Fingerprint 79A2 9CA4 6CC4 45DD > A9= > 04 =C2=A0C70E E654 3BB2 FA62 B9F1=C2=A0</div></div></div> > > --0000000000000e664e059947d3b2-- > > ------------------------------ > > Date: Mon, 9 Dec 2019 11:54:10 -0500 > From: Frank R <frankaritchie(a)gmail.com> > Subject: [ceph-users] ceph mgr daemon multiple ip addresses > To: ceph-users <ceph-users(a)ceph.com> > Message-ID: > <CAMuVLDOhax+8RwPEK_52UfzmOc8dv5= > YBH7Etsm5VmiO_WFEPg(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="0000000000004ef36d0599483fd9" > > --0000000000004ef36d0599483fd9 > Content-Type: text/plain; charset="UTF-8" > > Hi all, > > Does anyone know what possible issues can arise if the ceph mgr daemon is > running on a mon node that has 2 ips in the public net range (1 is a > loopback address). > > As I understand the it. mgr will bind to all ips > > FYI - I am not sure why the loopback is there, I am trying to find out. > > thx > Frank > > > > > mlovell - ceph anycast > > --0000000000004ef36d0599483fd9 > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"ltr"><div>Hi all,</div><div><br>Does anyone know what possible > = > issues can arise if the ceph mgr daemon is running on a mon node that has > 2= > ips in the public net range (1 is a loopback > address).<br><br></div><div>A= > s I understand the it. mgr will bind to all ips<br><br></div><div>FYI - I > a= > m not sure why the loopback is there, I am trying to find > out.</div><div><b= > r></div><div>thx</div><div>Frank<br><br><br><br><br>mlovell - ceph > anycast<= > br></div></div> > > --0000000000004ef36d0599483fd9-- > > ------------------------------ > > Date: Mon, 9 Dec 2019 17:24:27 +0000 > From: Bryan Stillwell <bstillwell(a)godaddy.com> > Subject: [ceph-users] Re: osdmaps not trimmed until ceph-mon's > restarted (if cluster has a down osd) > To: Dan van der Ster <dan(a)vanderster.com> > Cc: Joao Eduardo Luis <joao(a)suse.de>, "dev(a)ceph.io" <dev(a)ceph.io>, > ceph-users <ceph-users(a)ceph.io> > Message-ID: <9B145B17-6665-4254-8D6A-04A9B37389C3(a)godaddy.com> > Content-Type: text/plain; charset="us-ascii" > > On Nov 18, 2019, at 8:12 AM, Dan van der Ster <dan(a)vanderster.com> wrote: > > > > On Fri, Nov 15, 2019 at 4:45 PM Joao Eduardo Luis <joao(a)suse.de> wrote: > >> > >> On 19/11/14 11:04AM, Gregory Farnum wrote: > >>> On Thu, Nov 14, 2019 at 8:14 AM Dan van der Ster <dan(a)vanderster.com> > wrote: > >>>> > >>>> Hi Joao, > >>>> > >>>> I might have found the reason why several of our clusters (and maybe > >>>> Bryan's too) are getting stuck not trimming osdmaps. > >>>> It seems that when an osd fails, the min_last_epoch_clean gets stuck > >>>> forever (even long after HEALTH_OK), until the ceph-mons are > >>>> restarted. > >>>> > >>>> I've updated the ticket: https://tracker.ceph.com/issues/41154 > >>> > >>> Wrong ticket, I think you meant > https://tracker.ceph.com/issues/37875#note-7 > >> > >> I've seen this behavior a long, long time ago, but stopped being able to > >> reproduce it consistently enough to ensure the patch was working > properly. > >> > >> I think I have a patch here: > >> > >> https://github.com/ceph/ceph/pull/19076/commits > >> > >> If you are feeling adventurous, and want to give it a try, let me know. > I'll > >> be happy to forward port it to whatever you are running. > > > > Thanks Joao, this patch is what I had in mind. > > > > I'm trying to evaluate how adventurous this would be -- Is there any > > risk that if a huge number of osds are down all at once (but > > transiently), it would trigger the mon to trim too many maps? > > I would expect that the remaining up OSDs will have a safe, low, > osd_epoch ? > > > > And anyway I guess that your proposed get_min_last_epoch_clean patch > > is equivalent to what we have today if we restart the ceph-mon leader > > while an osd is down. > > Joao, > > I ran into this again today and found over 100,000 osdmaps on all 1,000 > OSDs (~50 TiB of disk space used just by osdmaps). There were down OSDs > (pretty regular occurrence with ~1,000 OSDs) so that matches up with what > Dan found. Then when I restarted all the mon nodes twice the osdmaps > started cleaning up. > > I believe the steps to reproduce would look like this: > > 1. Start with a cluster with at least 1 down osd > 2. Expand the cluster (the bigger the expansion, the more osdmaps that > pile up) > 3. Notice that after the expansion completes and the cluster is healthy > that the old osdmaps aren't cleaned up > > I would be willing to test the fix on our test cluster after 14.2.5 comes > out. Could you make a build based on that release? > > Thanks, > Bryan > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 83, Issue 35 > ****************************************** > -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 0

Re: : RGW listing millions of objects takes too much time
by Romit Misra 10 Dec '19

10 Dec '19

Hi Robert and Arash, A couple of pointers and asks that might help. 1. Can you point to the code you are using for listing the buckets. 2. Which release is the cluster running on..? 3. What is the number of shards that have been configured in the bucket index for the said mentioned bucket...? 4.Have you tried timing the listing process, as to where does the slow down began, in other words after 1 million entries, 2 million entri s, or the listing is slow at per key level. 5. You could profile the timing for every 10000 entries or 100000 6.While using a additional meta, like redis in memory meta store could be a solution, but moving many moving parts altogether becomes a headache. 7. Let's look at the tunebales first, to see where the bottleneck is. 8.Another good practice , of possible in your deployment would be, if you can restrict objects per bucket, to say .5 million or 1 million, and have a client side hash/placement algorithm for writing into the buckets. 9. If these 30 million entries are static, iff , then prefetching the markers , and running are multithreaded flow makes sense. But again if these are static why would you not store them in a blob altogether, so contraindicating my own point. 10.Also do look ceph lists and mailers for ordered Vs unordered listing(remember seeing some query around it) Thanks Romit Misra On Mon, 9 Dec 2019, 22:56 , <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. RGW listing millions of objects takes too much time (Arash Shams) > 2. Re: RGW listing millions of objects takes too much time > (Robert LeBlanc) > 3. ceph mgr daemon multiple ip addresses (Frank R) > 4. Re: osdmaps not trimmed until ceph-mon's restarted (if cluster has a > down osd) > (Bryan Stillwell) > > > ---------------------------------------------------------------------- > > Date: Mon, 9 Dec 2019 15:46:04 +0000 > From: Arash Shams <ara4sh(a)hotmail.com> > Subject: [ceph-users] RGW listing millions of objects takes too much > time > To: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <CWXP265MB128679DE9E6B2A6CA289B16492580(a)CWXP265MB1286.GBR > P265.PROD.OUTLOOK.COM> > Content-Type: multipart/alternative; boundary="_000_CWXP265MB128679 > DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_" > > --_000_CWXP265MB128679DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_ > Content-Type: text/plain; charset="iso-8859-1" > Content-Transfer-Encoding: quoted-printable > > Dear All, > > I have almost 30 million objects and I want to list them and index them > som= > ewhere else, > Im using boto3 with continuation Marker but it takes almost 9 hours > > can I run it in multiple threads to make it faster? what solution do you > su= > ggest to speedup this process, > > > Thanks > > > --_000_CWXP265MB128679DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_ > Content-Type: text/html; charset="iso-8859-1" > Content-Transfer-Encoding: quoted-printable > > <html> > <head> > <meta http-equiv=3D"Content-Type" content=3D"text/html; > charset=3Diso-8859-= > 1"> > <style type=3D"text/css" style=3D"display:none;"> P > {margin-top:0;margin-bo= > ttom:0;} </style> > </head> > <body dir=3D"ltr"> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > Dear All, </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > I have almost 30 million objects and I want to list them and index them > som= > ewhere else, </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > Im using boto3 with continuation Marker but it takes almost 9 hours</div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > can I run it in multiple threads to make it faster? what solution do you > su= > ggest to speedup this process, </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > Thanks </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > </body> > </html> > > --_000_CWXP265MB128679DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_-- > > ------------------------------ > > Date: Mon, 9 Dec 2019 08:23:55 -0800 > From: Robert LeBlanc <robert(a)leblancnet.us> > Subject: [ceph-users] Re: RGW listing millions of objects takes too > much time > To: Arash Shams <ara4sh(a)hotmail.com> > Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: > <CAANLjFoZu8LE0eH2vB9QArAOZJf4Ofm8V= > jAW-+ZPOFS-f9Oag(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="0000000000000e664e059947d3b2" > > --0000000000000e664e059947d3b2 > Content-Type: text/plain; charset="UTF-8" > > On Mon, Dec 9, 2019 at 7:47 AM Arash Shams <ara4sh(a)hotmail.com> wrote: > > > Dear All, > > > > I have almost 30 million objects and I want to list them and index them > > somewhere else, > > Im using boto3 with continuation Marker but it takes almost 9 hours > > > > can I run it in multiple threads to make it faster? what solution do you > > suggest to speedup this process, > > > > > > Thanks > > > > I've thought about indexing objects elsewhere as well. One thought I had > was hooking into the HTTP flow where a PUT or DEL would update the objects > in some kind of database (async of course). We could also gather stats with > GET and POST. Initially, my thoughts were to hook into haproxy since we > already use it, but possibly RGW if that is an option. That way it would > always be up to date and not have to do big scans on the buckets (our > buckets would not perform well with this). I haven't actually gotten to the > implementation phase of this idea. > > ---------------- > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > > --0000000000000e664e059947d3b2 > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"ltr"><div dir=3D"ltr">On Mon, Dec 9, 2019 at 7:47 AM Arash > Sham= > s <<a href=3D"mailto:ara4sh@hotmail.com">ara4sh(a)hotmail.com</a>> > wrot= > e:<br></div><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" > st= > yle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid > rgb(204,204,204);padd= > ing-left:1ex"> > > > > > <div dir=3D"ltr"> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > Dear All,=C2=A0</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > I have almost 30 million objects and I want to list them and index them > som= > ewhere else,=C2=A0</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > Im using boto3 with continuation Marker but it takes almost 9 hours</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > can I run it in multiple threads to make it faster? what solution do you > su= > ggest to speedup this process,=C2=A0</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > Thanks=C2=A0</div></div></blockquote><div > class=3D"gmail_quote"><br></div><= > div class=3D"gmail_quote">I've thought about indexing objects > elsewhere= > as well. One thought I had was hooking into the HTTP flow where a PUT or > D= > EL would update the objects in some kind of database (async of course). We > = > could also gather stats with GET and POST. Initially, my thoughts were to > h= > ook into haproxy since we already use it, but possibly RGW if that is an > op= > tion. That way it would always be up to date and not have to do big scans > o= > n the buckets (our buckets would not perform well with this). I > haven't= > actually gotten to the implementation phase of this idea.</div><br clear= > =3D"all"><div><div dir=3D"ltr" > class=3D"gmail_signature">----------------<b= > r>Robert LeBlanc<br></div></div><div>PGP Fingerprint 79A2 9CA4 6CC4 45DD > A9= > 04 =C2=A0C70E E654 3BB2 FA62 B9F1=C2=A0</div></div></div> > > --0000000000000e664e059947d3b2-- > > ------------------------------ > > Date: Mon, 9 Dec 2019 11:54:10 -0500 > From: Frank R <frankaritchie(a)gmail.com> > Subject: [ceph-users] ceph mgr daemon multiple ip addresses > To: ceph-users <ceph-users(a)ceph.com> > Message-ID: > <CAMuVLDOhax+8RwPEK_52UfzmOc8dv5= > YBH7Etsm5VmiO_WFEPg(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="0000000000004ef36d0599483fd9" > > --0000000000004ef36d0599483fd9 > Content-Type: text/plain; charset="UTF-8" > > Hi all, > > Does anyone know what possible issues can arise if the ceph mgr daemon is > running on a mon node that has 2 ips in the public net range (1 is a > loopback address). > > As I understand the it. mgr will bind to all ips > > FYI - I am not sure why the loopback is there, I am trying to find out. > > thx > Frank > > > > > mlovell - ceph anycast > > --0000000000004ef36d0599483fd9 > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"ltr"><div>Hi all,</div><div><br>Does anyone know what possible > = > issues can arise if the ceph mgr daemon is running on a mon node that has > 2= > ips in the public net range (1 is a loopback > address).<br><br></div><div>A= > s I understand the it. mgr will bind to all ips<br><br></div><div>FYI - I > a= > m not sure why the loopback is there, I am trying to find > out.</div><div><b= > r></div><div>thx</div><div>Frank<br><br><br><br><br>mlovell - ceph > anycast<= > br></div></div> > > --0000000000004ef36d0599483fd9-- > > ------------------------------ > > Date: Mon, 9 Dec 2019 17:24:27 +0000 > From: Bryan Stillwell <bstillwell(a)godaddy.com> > Subject: [ceph-users] Re: osdmaps not trimmed until ceph-mon's > restarted (if cluster has a down osd) > To: Dan van der Ster <dan(a)vanderster.com> > Cc: Joao Eduardo Luis <joao(a)suse.de>, "dev(a)ceph.io" <dev(a)ceph.io>, > ceph-users <ceph-users(a)ceph.io> > Message-ID: <9B145B17-6665-4254-8D6A-04A9B37389C3(a)godaddy.com> > Content-Type: text/plain; charset="us-ascii" > > On Nov 18, 2019, at 8:12 AM, Dan van der Ster <dan(a)vanderster.com> wrote: > > > > On Fri, Nov 15, 2019 at 4:45 PM Joao Eduardo Luis <joao(a)suse.de> wrote: > >> > >> On 19/11/14 11:04AM, Gregory Farnum wrote: > >>> On Thu, Nov 14, 2019 at 8:14 AM Dan van der Ster <dan(a)vanderster.com> > wrote: > >>>> > >>>> Hi Joao, > >>>> > >>>> I might have found the reason why several of our clusters (and maybe > >>>> Bryan's too) are getting stuck not trimming osdmaps. > >>>> It seems that when an osd fails, the min_last_epoch_clean gets stuck > >>>> forever (even long after HEALTH_OK), until the ceph-mons are > >>>> restarted. > >>>> > >>>> I've updated the ticket: https://tracker.ceph.com/issues/41154 > >>> > >>> Wrong ticket, I think you meant > https://tracker.ceph.com/issues/37875#note-7 > >> > >> I've seen this behavior a long, long time ago, but stopped being able to > >> reproduce it consistently enough to ensure the patch was working > properly. > >> > >> I think I have a patch here: > >> > >> https://github.com/ceph/ceph/pull/19076/commits > >> > >> If you are feeling adventurous, and want to give it a try, let me know. > I'll > >> be happy to forward port it to whatever you are running. > > > > Thanks Joao, this patch is what I had in mind. > > > > I'm trying to evaluate how adventurous this would be -- Is there any > > risk that if a huge number of osds are down all at once (but > > transiently), it would trigger the mon to trim too many maps? > > I would expect that the remaining up OSDs will have a safe, low, > osd_epoch ? > > > > And anyway I guess that your proposed get_min_last_epoch_clean patch > > is equivalent to what we have today if we restart the ceph-mon leader > > while an osd is down. > > Joao, > > I ran into this again today and found over 100,000 osdmaps on all 1,000 > OSDs (~50 TiB of disk space used just by osdmaps). There were down OSDs > (pretty regular occurrence with ~1,000 OSDs) so that matches up with what > Dan found. Then when I restarted all the mon nodes twice the osdmaps > started cleaning up. > > I believe the steps to reproduce would look like this: > > 1. Start with a cluster with at least 1 down osd > 2. Expand the cluster (the bigger the expansion, the more osdmaps that > pile up) > 3. Notice that after the expansion completes and the cluster is healthy > that the old osdmaps aren't cleaned up > > I would be willing to test the fix on our test cluster after 14.2.5 comes > out. Could you make a build based on that release? > > Thanks, > Bryan > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 83, Issue 35 > ****************************************** > -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 0

High swap usage on one replication node
by Götz Reinicke 09 Dec '19

09 Dec '19

Hi, our Ceph 14.2.3 cluster so far runs smooth with replicated and EC pools, but since a couple of days one of the dedicated replication nodes consumes up to 99% swap and stays at that level. The other two replicated nodes use +- 50 - 60% of swap. All the 24 NVMe OSDs per node are BlueStore with default settings, 128GB RAM. The vm.swappiness is set to 10. Do you have any suggestions how to handle/reduce the swap usage? Thanks for feedback and regards . Götz

5 5

ceph mgr daemon multiple ip addresses
by Frank R 09 Dec '19

09 Dec '19

Hi all, Does anyone know what possible issues can arise if the ceph mgr daemon is running on a mon node that has 2 ips in the public net range (1 is a loopback address). As I understand the it. mgr will bind to all ips FYI - I am not sure why the loopback is there, I am trying to find out. thx Frank mlovell - ceph anycast

2 1