- ceph-users - lists.ceph.io

Nautilus: pg_autoscaler causes mon slow ops
by Eugen Block 17 Sep '19

17 Sep '19

Hi everyone, we upgraded our production cluster just recently to version: ceph version 14.2.3-349-g7b1552ea82 (7b1552ea827cf5167b6edbba96dd1c4a9dc16937) nautilus (stable) We then activated pg_autoscaler for two pools that had a bad pg_num and the result is satisfying. However, after the rebalance finished the cluster became laggy. We noticed that two out of three MONs had a much higher CPU usage than usual, according to `top` the MON processes consumed more than 100%. Restarting the MON services and disabling pg_autoscaler resolved the issue. I've read that the balancer module can cause a higher load on the MGR daemon, is this somehow related? Another thing to mention is the confusing calculation of the autoscaler. After the pg numbers had been corrected we got the warning about overcommitted pools: > 1 subtrees have overcommitted pool target_size_bytes > 1 subtrees have overcommitted pool target_size_ratio The images pool was responsible for that. The confusing part was that sometimes autoscale-status displayed the size of that pool with more than 14 TB: ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE images 14399G 3.0 33713G 1.2813 1.0 128 on And a couple of minutes later the pool suddenly only had around 4 TB of data: ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE images 4112G 3.0 33713G 0.3659 1.0 128 on There seems to be some kind of inconsistency here. The actual used storage of this pool according to `ceph df` is: POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL images 1 4.1 TiB 1.01M 12 TiB 49.73 4.1 TiB Has anyone experienced something similar? Are these known issues? Regards, Eugen

1 0

Intergrate Metadata with ElasticSeach
by tuan dung 17 Sep '19

17 Sep '19

Dear all, Can you show me steps how to intergrate Metadata of Ceph object with ElasticSeach to improve searching medata performance? thank you very much. ----------------------------- Br, Dương Tuấn Dũng 0986153686

1 0

Fwd: ceph-users Digest, Vol 80, Issue 54
by Rom Freiman 16 Sep '19

16 Sep '19

unsubscribe ---------- Forwarded message --------- From: <ceph-users-request(a)ceph.io> Date: Mon, Sep 16, 2019 at 7:22 PM Subject: ceph-users Digest, Vol 80, Issue 54 To: <ceph-users(a)ceph.io> Send ceph-users mailing list submissions to ceph-users(a)ceph.io To subscribe or unsubscribe via email, send a message with subject or body 'help' to ceph-users-request(a)ceph.io You can reach the person managing the list at ceph-users-owner(a)ceph.io When replying, please edit your Subject line so it is more specific than "Re: Contents of ceph-users digest..." Today's Topics: 1. Re: upmap supported in SLES 12SPx (Ilya Dryomov) 2. Re: upmap supported in SLES 12SPx (Thomas Schneider) 3. Re: upmap supported in SLES 12SPx (Ilya Dryomov) 4. Re: Using same instance name for rgw (Eric Choi) 5. Re: RGW Passthrough (Casey Bodley) ---------------------------------------------------------------------- Date: Mon, 16 Sep 2019 16:56:19 +0200 From: Ilya Dryomov <idryomov(a)gmail.com> Subject: [ceph-users] Re: upmap supported in SLES 12SPx To: Thomas Schneider <74cmonty(a)gmail.com> Cc: Konstantin Shalygin <k0ste(a)k0ste.ru>, ceph-users <ceph-users(a)ceph.io> Message-ID: <CAOi1vP-YzuMeL-u=hPDMr6fSTWd+F7RXX61kfO9609pxv98qNw(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" On Mon, Sep 16, 2019 at 4:40 PM Thomas Schneider <74cmonty(a)gmail.com> wrote: > > Hi, > > thanks for your valuable input. > > Question: > Can I get more information of the 6 clients (those with features > 0x40106b84a842a42), e.g. IP, that allows me to identify it easily? Yes, although it's not integrated into "ceph features". Log into a monitor node and run "ceph daemon mon.a sessions" (mon.a is the name of the monitor, substitute accordingly). Thanks, Ilya ------------------------------ Date: Mon, 16 Sep 2019 17:10:37 +0200 From: Thomas Schneider <74cmonty(a)gmail.com> Subject: [ceph-users] Re: upmap supported in SLES 12SPx To: Ilya Dryomov <idryomov(a)gmail.com> Cc: Konstantin Shalygin <k0ste(a)k0ste.ru>, ceph-users <ceph-users(a)ceph.io> Message-ID: <79b342ac-be00-e3e7-cbec-b6c96d3a0a59(a)gmail.com> Content-Type: text/plain; charset=utf-8 Wonderbra. I found some relevant sessions on 2 of 3 monitor nodes. And I found some others: root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42 root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous [ "MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52 (jewel))", ] Would it make sense to shutdown these clients, too? What confuses me is that the list includes clients that belong to the Ceph cluster, namely 10.97.206.0/24. All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph. Regards Thomas Am 16.09.2019 um 16:56 schrieb Ilya Dryomov: > On Mon, Sep 16, 2019 at 4:40 PM Thomas Schneider <74cmonty(a)gmail.com> wrote: >> Hi, >> >> thanks for your valuable input. >> >> Question: >> Can I get more information of the 6 clients (those with features >> 0x40106b84a842a42), e.g. IP, that allows me to identify it easily? > Yes, although it's not integrated into "ceph features". Log into > a monitor node and run "ceph daemon mon.a sessions" (mon.a is the name > of the monitor, substitute accordingly). > > Thanks, > > Ilya ------------------------------ Date: Mon, 16 Sep 2019 17:36:35 +0200 From: Ilya Dryomov <idryomov(a)gmail.com> Subject: [ceph-users] Re: upmap supported in SLES 12SPx To: Thomas Schneider <74cmonty(a)gmail.com> Cc: Konstantin Shalygin <k0ste(a)k0ste.ru>, ceph-users <ceph-users(a)ceph.io> Message-ID: <CAOi1vP9MsfoQFPNfUYPRn2MzS=5buZ-s2Jcorkym84K8hJ6cWw(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" On Mon, Sep 16, 2019 at 5:10 PM Thomas Schneider <74cmonty(a)gmail.com> wrote: > > Wonderbra. > > I found some relevant sessions on 2 of 3 monitor nodes. > And I found some others: > root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42 > root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous > [ > "MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open > allow *, features 0x27018fb86aa42ada (jewel))", > "MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open > allow *, features 0x27018fb86aa42ada (jewel))", > "MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open > allow r, features 0x27018eb84aa42a52 (jewel))", > "MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open > allow r, features 0x27018fb86aa42ada (jewel))", > "MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open > allow r, features 0x27018fb86aa42ada (jewel))", > "MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open > allow *, features 0x27018fb86aa42ada (jewel))", > "MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open > allow r, features 0x27018fb86aa42ada (jewel))", > "MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow > r, features 0x27018eb84aa42a52 (jewel))", > "MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open > allow r, features 0x27018eb84aa42a52 (jewel))", > "MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open > allow r, features 0x27018eb84aa42a52 (jewel))", > "MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open > allow r, features 0x27018fb86aa42ada (jewel))", > "MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow > r, features 0x27018fb86aa42ada (jewel))", > "MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52 > (jewel))", > ] > > Would it make sense to shutdown these clients, too? > > What confuses me is that the list includes clients that belong to the > Ceph cluster, namely 10.97.206.0/24. > All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph. The above output seems consistent with your "ceph features" output: it lists clients with features 0x27018eb84aa42a52 and 0x27018fb86aa42ada. Like I said in my previous email, both of these support upmap. If you temporarily shut them down, set-require-min-compat-client will work without --yes-i-really-mean-it. Thanks, Ilya ------------------------------ Date: Mon, 16 Sep 2019 16:05:47 -0000 From: "Eric Choi" <eric.yongjun.choi(a)gmail.com> Subject: [ceph-users] Re: Using same instance name for rgw To: ceph-users(a)ceph.io Message-ID: <156864994756.18.7543223789861447910@mailman-web> Content-Type: text/plain; charset="utf-8" bump. anyone? ------------------------------ Date: Mon, 16 Sep 2019 12:22:16 -0400 From: Casey Bodley <cbodley(a)redhat.com> Subject: [ceph-users] Re: RGW Passthrough To: ceph-users(a)ceph.io Message-ID: <fdf93930-794e-b2d0-46ee-5288a4d91605(a)redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Hi Robert, So far the cloud tiering features are still in the design stages. We're working on some initial refactoring work to support this abstraction (ie. to either satisfy a request against the local rados cluster, or to proxy it somewhere else). With respect to passthrough/tiering to AWS, we could use help thinking through the user/credential mapping in particular. We have a weekly 'RGW Refactoring' meeting on Wednesdays where we discuss design and refactoring progress - it's on the upstream community calendar, I'll send you an invite. On 9/13/19 9:59 PM, Robert LeBlanc wrote: > We are very interested in the RGW Passthrough mentioned for Octupus. > What's the status and how can we help? We want to connect with AWS S3. > > Thank you, > ---------------- > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io ------------------------------ Subject: Digest Footer _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s ------------------------------ End of ceph-users Digest, Vol 80, Issue 54 ******************************************

1 0

RGW Passthrough
by Robert LeBlanc 16 Sep '19

16 Sep '19

We are very interested in the RGW Passthrough mentioned for Octupus. What's the status and how can we help? We want to connect with AWS S3. Thank you, ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 1

Using same instance name for rgw
by Eric Choi 16 Sep '19

16 Sep '19

I previously posted this question to lists.ceph.com not understanding lists.ceph.io is the replacement for it. Posting it again here with some edits. --- Hi there, we have been using ceph for a few years now, it's only now that I've noticed we have been using the same name for all RGW hosts, resulting when you run ceph -s: rgw: 1 daemon active (..) also our ceph.conf looks like (for rgw) ... [client.radosgw.gateway] ... despite having more than 10 RGW hosts. * What are the side effects of doing this? Is this a no-no? I can see the metrics can (ceph daemon ... perf dump) be wrong, are the metrics kept track independently (per host)? Can this affect performance negatively by any chance? (we are using the same key obviously..) * We recently upgraded from Lumunious to Nautilus, I've noticed that later docs all prescribes radosgw config section as [client.rgw.{instance-name}] .. should we make this change? Much appreciated!

1 1

Different pools count in ceph -s and ceph osd pool ls
by Fyodor Ustinov 16 Sep '19

16 Sep '19

Hi! I create bug https://tracker.ceph.com/issues/41832 Maybe someone also encountered such a problem? WBR, Fyodor.

1 0

Re: Ceph Day London - October 24 (Call for Papers!)
by Wido den Hollander 16 Sep '19

16 Sep '19

Hi, The CFP is ending today for the Ceph Day London on October 24th. If you have a talk you would like to submit, please follow the link below! Wido On 7/18/19 3:43 PM, Wido den Hollander wrote: > Hi, > > We will be having Ceph Day London October 24th! > > https://ceph.com/cephdays/ceph-day-london-2019/ > > The CFP is now open for you to get your Ceph related content in front > of the Ceph community ranging from all levels of expertise: > > https://forms.zohopublic.com/thingee/form/CephDayLondon2019/formperma/h96jZ… > > If your company is interested in sponsoring the event, we would be > delighted to have you. Please contact me directly for further information. > > The Ceph Day is co-located with the Apache CloudStack project. There > will be two tracks where people can choose between Ceph and CloudStack. > > After the Ceph Day there's going to be beers in the pub nearby to make > new friends. > > Join us in London on October 24th! > > Wido > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

1 0

Warning: 1 pool nearfull and unbalanced data distribution
by Thomas 16 Sep '19

16 Sep '19

Update: I have been pointed to this <https://docs.ceph.com/docs/master/rados/operations/upmap/> doku "Using the pg-upmap" and I checked the offline optimization. However the output for the relevant pool shows: no upmaps proposed root@ld3955:/mnt/rbd# osdmaptool om --upmap out.txt --upmap-pool hdb_backup osdmaptool: osdmap file 'om' writing upmap command output to: out.txt checking for upmap cleanups upmap, max-count 100, max deviation 0.01 limiting to pools hdb_backup (11) no upmaps proposed This is very strange because it is clear that this pool hdb_backup is very much unbalanced. Please advise. THX .......................................... Hi, the output of ceph health details gives me a warning that concerns me a little. I'll explain in a second. root@ld3955:/mnt/rbd# ceph health detail HEALTH_WARN 1 nearfull osd(s); 1 pool(s) nearfull; 4 pools have too many placement groups OSD_NEARFULL 1 nearfull osd(s) osd.122 is near full POOL_NEARFULL 1 pool(s) nearfull pool 'hdb_backup' is nearfull POOL_TOO_MANY_PGS 4 pools have too many placement groups Pool pve_cephfs_data has 128 placement groups, should have 16 Pool hdd has 512 placement groups, should have 64 Pool pve_cephfs_metadata has 32 placement groups, should have 4 Pool backup has 1024 placement groups, should have 4 I'm writing +90% of the data in pool "hdb_backup", and this is ongoing. Therefore I can hardly afford that this pool is full. When I check the output of ceph osd status the relevand osd is somehow overutilized: root@ld3955:/mnt/rbd# ceph osd status | grep nearfull +-----+--------+-------+-------+--------+---------+--------+---------+--------------------+ | id | host | used | avail | wr ops | wr data | rd ops | rd data | state | +-----+--------+-------+-------+--------+---------+--------+---------+--------------------+ | 122 | ld5505 | 1448G | 227G | 0 | 0 | 0 | 0 | exists,nearfull,up | This looks like an inconsistency, but I can check the usage with ceph osd df. Here I can see that the OSDs are not really balanced: root@ld3955:/mnt/rbd# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 272 hdd 7.28000 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 136 KiB 3.7 GiB 5.0 TiB 31.23 0.77 121 up 273 hdd 7.28000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 28 KiB 5.1 GiB 4.4 TiB 39.35 0.97 152 up 274 hdd 7.28000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 168 KiB 4.5 GiB 4.4 TiB 39.41 0.97 152 up 275 hdd 7.28000 1.00000 7.3 TiB 2.2 TiB 2.2 TiB 139 KiB 3.5 GiB 5.1 TiB 29.77 0.73 115 up 276 hdd 7.28000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 48 KiB 5.7 GiB 4.5 TiB 38.81 0.96 150 up 277 hdd 7.28000 1.00000 7.3 TiB 2.5 TiB 2.5 TiB 276 KiB 4.3 GiB 4.8 TiB 34.68 0.85 134 up 278 hdd 7.28000 1.00000 7.3 TiB 2.8 TiB 2.8 TiB 36 KiB 4.4 GiB 4.5 TiB 38.74 0.95 150 up 279 hdd 7.28000 1.00000 7.3 TiB 2.6 TiB 2.6 TiB 156 KiB 4.1 GiB 4.7 TiB 35.80 0.88 138 up 280 hdd 7.28000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 156 KiB 4.3 GiB 4.6 TiB 37.03 0.91 143 up 281 hdd 7.28000 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 172 KiB 3.8 GiB 4.9 TiB 32.67 0.80 126 up 282 hdd 7.28000 1.00000 7.3 TiB 2.9 TiB 2.9 TiB 120 KiB 4.5 GiB 4.4 TiB 39.39 0.97 152 up 283 hdd 7.28000 1.00000 7.3 TiB 2.7 TiB 2.7 TiB 32 KiB 5.9 GiB 4.5 TiB 37.57 0.93 145 up [...] 76 hdd 1.64000 1.00000 1.6 TiB 1.4 TiB 1.4 TiB 88 KiB 2.4 GiB 268 GiB 84.02 2.07 73 up 77 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 188 KiB 2.0 GiB 560 GiB 66.59 1.64 58 up 78 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1023 GiB 164 KiB 1.9 GiB 651 GiB 61.15 1.51 53 up 79 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 176 KiB 1.9 GiB 636 GiB 62.02 1.53 54 up 80 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 80 KiB 2.5 GiB 636 GiB 62.07 1.53 54 up 81 hdd 1.64000 1.00000 1.6 TiB 886 GiB 885 GiB 128 KiB 1.7 GiB 790 GiB 52.89 1.30 46 up 82 hdd 1.64000 1.00000 1.6 TiB 967 GiB 965 GiB 240 KiB 1.8 GiB 709 GiB 57.70 1.42 50 up 83 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 64 KiB 2.2 GiB 420 GiB 74.94 1.85 65 up 84 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 108 KiB 2.0 GiB 597 GiB 64.37 1.59 56 up 85 hdd 1.64000 1.00000 1.6 TiB 811 GiB 810 GiB 176 KiB 1.6 GiB 865 GiB 48.42 1.19 42 up 86 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 72 KiB 2.0 GiB 613 GiB 63.43 1.56 55 up 87 hdd 1.64000 1.00000 1.6 TiB 791 GiB 789 GiB 68 KiB 1.6 GiB 885 GiB 47.17 1.16 41 up 88 hdd 1.64000 1.00000 1.6 TiB 908 GiB 906 GiB 168 KiB 1.8 GiB 768 GiB 54.18 1.33 47 up [...] 113 hdd 1.64000 1.00000 1.6 TiB 1.3 TiB 1.3 TiB 100 KiB 3.0 GiB 342 GiB 79.60 1.96 69 up 114 hdd 1.64000 1.00000 1.6 TiB 1001 GiB 999 GiB 184 KiB 1.9 GiB 675 GiB 59.70 1.47 52 up 115 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 120 KiB 2.2 GiB 407 GiB 75.70 1.86 66 up 116 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 92 KiB 2.0 GiB 597 GiB 64.39 1.59 56 up 117 hdd 1.64000 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 76 KiB 2.7 GiB 480 GiB 71.34 1.76 62 up 118 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 48 KiB 2.6 GiB 574 GiB 65.74 1.62 57 up 119 hdd 1.64000 1.00000 1.6 TiB 1.0 TiB 1.0 TiB 152 KiB 1.9 GiB 634 GiB 62.19 1.53 54 up 120 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 48 KiB 2.0 GiB 541 GiB 67.73 1.67 59 up 121 hdd 1.64000 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 48 KiB 2.0 GiB 556 GiB 66.82 1.65 58 up 122 hdd 1.64000 0.95001 1.6 TiB 1.4 TiB 1.4 TiB 184 KiB 2.5 GiB 227 GiB 86.44 2.13 75 up I assume that this is a result of my Ceph Cluster history, means I started with 4 OSD nodes with 48 drives @1.8TB. This was 345TB in total. I started to fill this storage up to 75%. Then I added 2 OSD nodes with 48 drives @8TB. I was not expecting that pool "hdb_backup" would be filled up in the near future. This is the ouput of ceph df: root@ld3955:/mnt/rbd# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.1 PiB 661 TiB 467 TiB 468 TiB 41.43 nvme 23 TiB 23 TiB 681 MiB 8.7 GiB 0.04 TOTAL 1.1 PiB 685 TiB 467 TiB 468 TiB 40.59 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL backup 4 0 B 0 0 B 0 35 TiB hdb_backup 11 154 TiB 40.39M 154 TiB 64.18 29 TiB hdd 30 1.1 TiB 281.21k 1.1 TiB 1.01 35 TiB pve_cephfs_data 32 318 GiB 91.83k 318 GiB 0.30 35 TiB pve_cephfs_metadata 33 155 MiB 61 155 MiB 0 35 TiB nvme 35 0 B 0 0 B 0 7.4 TiB Question: How can I start rebalancing data in order to have more data in the larger drives (8TB)? Or is it ok that the smaller drives (1.8TB) are filled by +60%? THX for your advice Thomas

1 0

Delete objects on large bucket very slow
by tuan dung 13 Sep '19

13 Sep '19

Dear Ceph team, I builde a object storage using ceph, a cluster with 5 nodes (9 sas10k 2TB (for data)+ 1 nvme (for metadata) in one node , using ceph version 14.2.3, nautilus (stable). My cluster storage over 100M (minlion) objects (files) with file's size ~ 50KB, and I want to delete them to free capacity and improve performance. When I delete object, speed delete object very slow, about ~ 35-36 objs/s at 100M (minlion) objects point, i using command: s3cmd del -r s3://mybucket/2019 to delete. Can you help me: - How does ceph's deleting object operator work? (it mean about how delete object flow in ceph works?) - how to improve speed of delete object operator ( objects/s) - how to delete mult - bestpractice for this usecase - other recommend.... Thank you very much. ----------------------------- Br, Dương Tuấn Dũng 0986153686

3 4

Legacy Bluestore Stats
by gaving＠voxtandem.com 13 Sep '19

13 Sep '19

I created a cluster on Nautilus 14.2.0, then upgraded to 14.2.1, and finally 14.2.3 Now I am seeing this warning that I thought should only appear if the cluster was created pre-Nautilus. Legacy BlueStore stats reporting detected on XX OSD I can't seem to find any information about this on a cluster that was always on 14.2 Gavin

1 0