- ceph-users - lists.ceph.io

Bucket policies with OpenStack integration and limiting access
by shubjero＠gmail.com 09 Sep '19

09 Sep '19

Good day, We have a Ceph cluster and make use of object-storage and integrate with OpenStack. Each OpenStack project/tenant is given a radosgw user which allows all keystone users of that project to access the object-storage as that single radosgw user. The radosgw user is the project id of the OpenStack project/tenant. Sometimes we have use cases where we want to access the object-storage outside of the swift-api and use tools like the aws-cli or homebrew java applications to access the object storage. For this use case what we do is generate S3 access/secret key for the specific radosgw user and they have full access to the object storage for that OpenStack project/tenant. What we want to know is if it is possible to provide granular access to containers within a single OpenStack project using S3 access keys or S3 sub-users? I know that the Swift API has ACL's that can limit by keystone user but we are exploring the possibility of doing this using S3 and S3 bucket policies so that the tools our team are developing (open source) are more transferrable to AWS S3 and Rados GW. Thanks all, Jared Baker Cloud Architect, OICR

1 0

Out of memory
by Sylvain PORTIER 09 Sep '19

09 Sep '19

Hi, On my ceph osd servers I have lot of "out of memory messages". My servers are configured with : - 32 G of memory - 11 HDD (3,5 T each) (+ 2 HDD for the system) And the error messages are : /[101292.017968] Out of memory: Kill process 2597 (ceph-osd) score 102 or sacrifice child// //[101292.018836] Killed process 2597 (ceph-osd) total-vm:5048008kB, anon-rss:3002648kB, file-rss:0kB, shmem-rss:0kB// / Top result : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 75469 ceph 20 0 4982324 3,988g 0 S 0,7 12,7 3:45.05 ceph-osd 75499 ceph 20 0 5095896 3,710g 0 S 0,3 11,8 4:08.93 ceph-osd 65848 ceph 20 0 5713748 3,329g 0 S 0,0 10,6 68:49.12 ceph-osd 67237 ceph 20 0 5580720 3,155g 0 S 0,0 10,1 57:15.99 ceph-osd 71113 ceph 20 0 5557608 3,101g 0 S 0,3 9,9 36:47.69 ceph-osd 74745 ceph 20 0 5117212 3,062g 0 S 3,7 9,8 8:13.56 ceph-osd 72494 ceph 20 0 5621156 2,828g 0 S 0,3 9,0 27:19.97 ceph-osd 70954 ceph 20 0 5765016 2,571g 0 S 0,3 8,2 40:02.00 ceph-osd 74817 ceph 20 0 5139328 2,510g 0 S 0,3 8,0 7:24.33 ceph-osd 76523 ceph 20 0 3324820 2,422g 0 S 0,3 7,7 0:55.54 ceph-osd Is there a way to limit or reduce the memory usage of each osd deamon ? Thank you, Regards, Sylvain PORTIER. --- L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast. https://www.avast.com/antivirus

4 5

How to test PG mapping with reweight
by Robert LeBlanc 07 Sep '19

07 Sep '19

I'd like to test how reweighting an OSD will change how the PGs map in the cluster. I suspect that I'd dump the CRUSH map and PGs in the cluster that I'm interested in then use osdmaptool. I'm not understanding how to use osdmaptool to set the reweight, then query a PG or the entire set of PGs that I'm interested in. I then suspect that if I'm okay with the new map that I could inject it into the cluster instead of having to run reweight on the OSD(s). This is a Jewel cluster and I'm trying to calculate OSD usage offline, then inject a map that is more distributed instead of doing a reweight, move the PGs which take a long time to just rinse and repeat over and over again. Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 3

Ceph client failed to mount RBD device after reboot
by Vang Le-Quy 06 Sep '19

06 Sep '19

I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1. We have too ceph client n01 and n03 with following software versions. # n01 ceph-common: Installed: 12.2.8-0ubuntu0.18.04.1 Candidate: 12.2.12-0ubuntu0.18.04.2 # n03 ceph-common: Installed: 12.2.11-0ubuntu0.18.04.2 Candidate: 12.2.12-0ubuntu0.18.04.2 # n01 uname -a Linux n01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux # n03 uname -a Linux n03 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Problem: N01 can mount its RBD after reboot whereas N03 can’t. To verify that it is something on client side, I upgrade n01 to have the same package version to n03 including ceph-common. And n01 does indeed fail to mount its CEPH RBD partition. Syslog shows messages like the following: Sep 6 12:52:33 n01 kernel: [ 875.160275] rbd: rbd0: no lock owners detected Sep 6 12:52:33 n01 kernel: [ 875.161069] rbd: rbd0: client2508785 seems dead, breaking lock Sep 6 12:52:33 n01 kernel: [ 875.161723] rbd: rbd0: blacklist of client2508785 failed: -13 Sep 6 12:52:33 n01 kernel: [ 875.161724] rbd: rbd0: failed to acquire lock: -13 Sep 6 12:52:33 n01 kernel: [ 875.161956] rbd: rbd0: no lock owners detected Sep 6 12:52:33 n01 kernel: [ 875.162755] rbd: rbd0: client2508785 seems dead, breaking lock Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: blacklist of client2508785 failed: -13 Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: failed to acquire lock: -13 Sep 6 12:52:33 n01 kernel: [ 875.163645] rbd: rbd0: no lock owners detected Sep 6 12:52:33 n01 kernel: [ 875.164431] rbd: rbd0: client2508785 seems dead, breaking lock Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: blacklist of client2508785 failed: -13 Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: failed to acquire lock: -13 Sep 6 12:52:33 n01 kernel: [ 875.165355] rbd: rbd0: no lock owners detected [signature_1482564062] Vang Quy Le Special Consultant in Data Science and Infrastructure T: (+45) 9940 7710 | Email: vle(a)its.aau.dk<mailto:vle@its.aau.dk> Kontor 0-1-91 | Selma Lagerløfs Vej 300 | DK-9220 Aalborg Ø |

4 5

Re: using non client.admin user for ceph-iscsi gateways
by Jason Dillaman 06 Sep '19

06 Sep '19

On Fri, Sep 6, 2019 at 12:00 PM Wesley Dillingham <wdillingham(a)godaddy.com> wrote: > > the iscsi-gateway.cfg seemingly allows for an alternative cephx user other than client.admin to be used, however the comments in the documentations says specifically to use client.admin. Hmm, can you point out where this is in the docs? Originally, tcmu-runner didn't support the ability to change the user id, but that has been available for about a year now [1]. > Other than having the cfg file point to the appropriate key/user with "gateway_keyring" and giving that client read caps on the mons and full access to the pool configured to be used for iscsi are any other particular steps / settings / actions needed? Just use "profile rbd" for your caps to keep it simple. > It seems prudent to not use client.admin but I don't want to have unstable behavior or untested setup. > > Thanks. > > Respectfully, > > Wes Dillingham > wdillingham(a)godaddy.com > Site Reliability Engineer IV - Platform Storage / Ceph > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io [1] https://github.com/open-iscsi/tcmu-runner/commit/c85ccdcfb7f4b17926eda1df89… -- Jason

2 1

using non client.admin user for ceph-iscsi gateways
by Wesley Dillingham 06 Sep '19

06 Sep '19

the iscsi-gateway.cfg seemingly allows for an alternative cephx user other than client.admin to be used, however the comments in the documentations says specifically to use client.admin. Other than having the cfg file point to the appropriate key/user with "gateway_keyring" and giving that client read caps on the mons and full access to the pool configured to be used for iscsi are any other particular steps / settings / actions needed? It seems prudent to not use client.admin but I don't want to have unstable behavior or untested setup. Thanks. Respectfully, Wes Dillingham wdillingham(a)godaddy.com Site Reliability Engineer IV - Platform Storage / Ceph

1 0

Followup: weird behaviour with ceph osd pool create and the "crush-rule" parameter (suddenly changes behaviour)
by aoanla＠gmail.com 06 Sep '19

06 Sep '19

So, whilst debugging the behaviour in the first thread I created, I needed to create and then destroy pools (to avoid running out of placement groups). So, I did something like: ceph osd pool create ec2pool 2048 2048 erasure glasgow-eci-test ec2pool 0 ceph osd pool create ec3pool 2048 2048 erasure glasgow-eci-test2 ec3pool 0 (for two different types of ecpool) and then removed them with ceph osd pool rm ec2pool ec2pool --yes-i-really-really-mean-it ceph osd pool rm ec3pool ec3pool --yes-i-really-really-mean-it Now, however, something seems to have broken, as if I attempt: ceph osd pool create ec4pool 2048 2048 erasure glasgow-eci-test3 ec4pool 0 it fails with Error ENOENT: specified rule ec4pool doesn't exist (which, of course, it does not, as the whole point of the syntax is that ceph should build the crush rule for me and name it appropriately; and this worked for all the previous times). ceph health returns HEALTH OK still. Any suggestions? I've googled around a bit on this, but I can't seem to find anyone discussing it... Sam

4 6

RGW bucket check --check-objects -fix failed
by EDH - Manuel Rios Fernandez 06 Sep '19

06 Sep '19

Hi, Were at 14.2.2 We just found a broken bucket index, trying to repair with the common commands ]# radosgw-admin bucket check --check-objects fix finish instantly, but bucket should have near 60-70TB info. [root@CEPH-MON01 home]# radosgw-admin bucket check --check-objects --bucket BUCKETNAME --debug_rgw=10 2019-09-05 20:08:13.908 7fdc8db37580 10 cannot find current period zonegroup using local zonegroup 2019-09-05 20:08:13.910 7fdc8db37580 10 Cannot find current period zone using local zone 2019-09-05 20:08:13.933 7fdc8db37580 2 all 8 watchers are set, enabling cache 2019-09-05 20:08:13.942 7fdc56ffd700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-09-05 20:08:13.948 7fdc8db37580 10 cache get: name=default.rgw.data.root++BUCKETNAME : miss 2019-09-05 20:08:13.950 7fdc8db37580 10 cache put: name=default.rgw.data.root++BUCKETNAME info.flags=0x16 2019-09-05 20:08:13.950 7fdc8db37580 10 adding default.rgw.data.root++BUCKETNAME to cache LRU end 2019-09-05 20:08:13.950 7fdc8db37580 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-09-05 20:08:13.950 7fdc8db37580 10 cache get: name=default.rgw.data.root++BUCKETNAME : type miss (requested=0x11, cached=0x16) 2019-09-05 20:08:13.950 7fdc8db37580 10 cache put: name=default.rgw.data.root++BUCKETNAME info.flags=0x11 2019-09-05 20:08:13.950 7fdc8db37580 10 moving default.rgw.data.root++BUCKETNAME to cache LRU end 2019-09-05 20:08:13.950 7fdc8db37580 10 cache get: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 : miss 2019-09-05 20:08:13.951 7fdc8db37580 10 cache put: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 info.flags=0x16 2019-09-05 20:08:13.951 7fdc8db37580 10 adding default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16 f590a82.16313306.1 to cache LRU end 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.acl bl.length()=145 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.lc bl.length()=467 2019-09-05 20:08:13.951 7fdc8db37580 10 cache get: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 : type miss (requested=0x13, cached=0x16) 2019-09-05 20:08:13.951 7fdc8db37580 10 cache put: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 info.flags=0x13 2019-09-05 20:08:13.951 7fdc8db37580 10 moving default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16 f590a82.16313306.1 to cache LRU end 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.acl bl.length()=145 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.lc bl.length()=467 2019-09-05 20:08:13.951 7fdc8db37580 10 chain_cache_entry: cache_locator=default.rgw.data.root++BUCKETNAME 2019-09-05 20:08:13.951 7fdc8db37580 10 chain_cache_entry: cache_locator=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4 fe0-bbe4-fdc16f590a82.16313306.1 2019-09-05 20:08:13.951 7fdc8db37580 10 cache get: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 : type miss (requested=0x16, cached=0x13) 2019-09-05 20:08:13.952 7fdc8db37580 10 cache put: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 info.flags=0x16 2019-09-05 20:08:13.952 7fdc8db37580 10 moving default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16 f590a82.16313306.1 to cache LRU end 2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr: name=user.rgw.acl bl.length()=145 2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr: name=user.rgw.lc bl.length()=467 2019-09-05 20:08:13.952 7fdc8db37580 10 cache get: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 : hit (requested=0x11, cached=0x17) 2019-09-05 20:08:13.952 7fdc8db37580 10 cls_bucket_list_ordered BUCKETNAME[48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3856921.7] start _multipart_[] num_entries 1001 2019-09-05 20:08:13.960 7fdc8db37580 2 removed watcher, disabling cache Any recomendations for forcé recreate bucket index? Regards Manuel

1 1

Heavily-linked lists.ceph.com pipermail archive now appears to lead to 404s
by Florian Haas 05 Sep '19

05 Sep '19

Hi, is there any chance the list admins could copy the pipermail archive from lists.ceph.com over to lists.ceph.io? It seems to contain an awful lot of messages referred elsewhere by their archive URL, many (all?) of which appear to now lead to 404s. Example: google "Set existing pools to use hdd device class only". The top hit is a link to http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/029078.html: $ curl -IL http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/029078.html HTTP/1.1 301 Moved Permanently Server: nginx/1.10.3 (Ubuntu) Date: Thu, 29 Aug 2019 12:48:13 GMT Content-Type: text/html Content-Length: 194 Connection: keep-alive Location: https://lists.ceph.io/pipermail/ceph-users-ceph.com/2018-August/029078.html Strict-Transport-Security: max-age=31536000 HTTP/1.1 404 Not Found Server: nginx Date: Thu, 29 Aug 2019 12:48:14 GMT Content-Type: text/html; charset=utf-8 Content-Length: 3774 Connection: keep-alive X-Frame-Options: SAMEORIGIN Vary: Accept-Language, Cookie Content-Language: en Or maybe this is just a redirect rule that needs to be cleverer or more specific, rather than the apparent catch-all .com/.io redirect? Cheers, Florian

5 7

disk failure
by solarflow99 05 Sep '19

05 Sep '19

One of the things i've come to notice is when HDD drives fail, they often recover in a short time and get added back to the cluster. This causes the data to rebalance back and forth, and if I set the noout flag I get a health warning. Is there a better way to avoid this?

4 6