- ceph-users - lists.ceph.io

How to test PG mapping with reweight
by Robert LeBlanc 07 Sep '19

07 Sep '19

I'd like to test how reweighting an OSD will change how the PGs map in the cluster. I suspect that I'd dump the CRUSH map and PGs in the cluster that I'm interested in then use osdmaptool. I'm not understanding how to use osdmaptool to set the reweight, then query a PG or the entire set of PGs that I'm interested in. I then suspect that if I'm okay with the new map that I could inject it into the cluster instead of having to run reweight on the OSD(s). This is a Jewel cluster and I'm trying to calculate OSD usage offline, then inject a map that is more distributed instead of doing a reweight, move the PGs which take a long time to just rinse and repeat over and over again. Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 3

Ceph client failed to mount RBD device after reboot
by Vang Le-Quy 06 Sep '19

06 Sep '19

I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1. We have too ceph client n01 and n03 with following software versions. # n01 ceph-common: Installed: 12.2.8-0ubuntu0.18.04.1 Candidate: 12.2.12-0ubuntu0.18.04.2 # n03 ceph-common: Installed: 12.2.11-0ubuntu0.18.04.2 Candidate: 12.2.12-0ubuntu0.18.04.2 # n01 uname -a Linux n01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux # n03 uname -a Linux n03 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Problem: N01 can mount its RBD after reboot whereas N03 can’t. To verify that it is something on client side, I upgrade n01 to have the same package version to n03 including ceph-common. And n01 does indeed fail to mount its CEPH RBD partition. Syslog shows messages like the following: Sep 6 12:52:33 n01 kernel: [ 875.160275] rbd: rbd0: no lock owners detected Sep 6 12:52:33 n01 kernel: [ 875.161069] rbd: rbd0: client2508785 seems dead, breaking lock Sep 6 12:52:33 n01 kernel: [ 875.161723] rbd: rbd0: blacklist of client2508785 failed: -13 Sep 6 12:52:33 n01 kernel: [ 875.161724] rbd: rbd0: failed to acquire lock: -13 Sep 6 12:52:33 n01 kernel: [ 875.161956] rbd: rbd0: no lock owners detected Sep 6 12:52:33 n01 kernel: [ 875.162755] rbd: rbd0: client2508785 seems dead, breaking lock Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: blacklist of client2508785 failed: -13 Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: failed to acquire lock: -13 Sep 6 12:52:33 n01 kernel: [ 875.163645] rbd: rbd0: no lock owners detected Sep 6 12:52:33 n01 kernel: [ 875.164431] rbd: rbd0: client2508785 seems dead, breaking lock Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: blacklist of client2508785 failed: -13 Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: failed to acquire lock: -13 Sep 6 12:52:33 n01 kernel: [ 875.165355] rbd: rbd0: no lock owners detected [signature_1482564062] Vang Quy Le Special Consultant in Data Science and Infrastructure T: (+45) 9940 7710 | Email: vle(a)its.aau.dk<mailto:vle@its.aau.dk> Kontor 0-1-91 | Selma Lagerløfs Vej 300 | DK-9220 Aalborg Ø |

4 5

Re: using non client.admin user for ceph-iscsi gateways
by Jason Dillaman 06 Sep '19

06 Sep '19

On Fri, Sep 6, 2019 at 12:00 PM Wesley Dillingham <wdillingham(a)godaddy.com> wrote: > > the iscsi-gateway.cfg seemingly allows for an alternative cephx user other than client.admin to be used, however the comments in the documentations says specifically to use client.admin. Hmm, can you point out where this is in the docs? Originally, tcmu-runner didn't support the ability to change the user id, but that has been available for about a year now [1]. > Other than having the cfg file point to the appropriate key/user with "gateway_keyring" and giving that client read caps on the mons and full access to the pool configured to be used for iscsi are any other particular steps / settings / actions needed? Just use "profile rbd" for your caps to keep it simple. > It seems prudent to not use client.admin but I don't want to have unstable behavior or untested setup. > > Thanks. > > Respectfully, > > Wes Dillingham > wdillingham(a)godaddy.com > Site Reliability Engineer IV - Platform Storage / Ceph > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io [1] https://github.com/open-iscsi/tcmu-runner/commit/c85ccdcfb7f4b17926eda1df89… -- Jason

2 1

using non client.admin user for ceph-iscsi gateways
by Wesley Dillingham 06 Sep '19

06 Sep '19

the iscsi-gateway.cfg seemingly allows for an alternative cephx user other than client.admin to be used, however the comments in the documentations says specifically to use client.admin. Other than having the cfg file point to the appropriate key/user with "gateway_keyring" and giving that client read caps on the mons and full access to the pool configured to be used for iscsi are any other particular steps / settings / actions needed? It seems prudent to not use client.admin but I don't want to have unstable behavior or untested setup. Thanks. Respectfully, Wes Dillingham wdillingham(a)godaddy.com Site Reliability Engineer IV - Platform Storage / Ceph

1 0

Followup: weird behaviour with ceph osd pool create and the "crush-rule" parameter (suddenly changes behaviour)
by aoanla＠gmail.com 06 Sep '19

06 Sep '19

So, whilst debugging the behaviour in the first thread I created, I needed to create and then destroy pools (to avoid running out of placement groups). So, I did something like: ceph osd pool create ec2pool 2048 2048 erasure glasgow-eci-test ec2pool 0 ceph osd pool create ec3pool 2048 2048 erasure glasgow-eci-test2 ec3pool 0 (for two different types of ecpool) and then removed them with ceph osd pool rm ec2pool ec2pool --yes-i-really-really-mean-it ceph osd pool rm ec3pool ec3pool --yes-i-really-really-mean-it Now, however, something seems to have broken, as if I attempt: ceph osd pool create ec4pool 2048 2048 erasure glasgow-eci-test3 ec4pool 0 it fails with Error ENOENT: specified rule ec4pool doesn't exist (which, of course, it does not, as the whole point of the syntax is that ceph should build the crush rule for me and name it appropriately; and this worked for all the previous times). ceph health returns HEALTH OK still. Any suggestions? I've googled around a bit on this, but I can't seem to find anyone discussing it... Sam

4 6

RGW bucket check --check-objects -fix failed
by EDH - Manuel Rios Fernandez 06 Sep '19

06 Sep '19

Hi, Were at 14.2.2 We just found a broken bucket index, trying to repair with the common commands ]# radosgw-admin bucket check --check-objects fix finish instantly, but bucket should have near 60-70TB info. [root@CEPH-MON01 home]# radosgw-admin bucket check --check-objects --bucket BUCKETNAME --debug_rgw=10 2019-09-05 20:08:13.908 7fdc8db37580 10 cannot find current period zonegroup using local zonegroup 2019-09-05 20:08:13.910 7fdc8db37580 10 Cannot find current period zone using local zone 2019-09-05 20:08:13.933 7fdc8db37580 2 all 8 watchers are set, enabling cache 2019-09-05 20:08:13.942 7fdc56ffd700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-09-05 20:08:13.948 7fdc8db37580 10 cache get: name=default.rgw.data.root++BUCKETNAME : miss 2019-09-05 20:08:13.950 7fdc8db37580 10 cache put: name=default.rgw.data.root++BUCKETNAME info.flags=0x16 2019-09-05 20:08:13.950 7fdc8db37580 10 adding default.rgw.data.root++BUCKETNAME to cache LRU end 2019-09-05 20:08:13.950 7fdc8db37580 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-09-05 20:08:13.950 7fdc8db37580 10 cache get: name=default.rgw.data.root++BUCKETNAME : type miss (requested=0x11, cached=0x16) 2019-09-05 20:08:13.950 7fdc8db37580 10 cache put: name=default.rgw.data.root++BUCKETNAME info.flags=0x11 2019-09-05 20:08:13.950 7fdc8db37580 10 moving default.rgw.data.root++BUCKETNAME to cache LRU end 2019-09-05 20:08:13.950 7fdc8db37580 10 cache get: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 : miss 2019-09-05 20:08:13.951 7fdc8db37580 10 cache put: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 info.flags=0x16 2019-09-05 20:08:13.951 7fdc8db37580 10 adding default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16 f590a82.16313306.1 to cache LRU end 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.acl bl.length()=145 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.lc bl.length()=467 2019-09-05 20:08:13.951 7fdc8db37580 10 cache get: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 : type miss (requested=0x13, cached=0x16) 2019-09-05 20:08:13.951 7fdc8db37580 10 cache put: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 info.flags=0x13 2019-09-05 20:08:13.951 7fdc8db37580 10 moving default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16 f590a82.16313306.1 to cache LRU end 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.acl bl.length()=145 2019-09-05 20:08:13.951 7fdc8db37580 10 updating xattr: name=user.rgw.lc bl.length()=467 2019-09-05 20:08:13.951 7fdc8db37580 10 chain_cache_entry: cache_locator=default.rgw.data.root++BUCKETNAME 2019-09-05 20:08:13.951 7fdc8db37580 10 chain_cache_entry: cache_locator=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4 fe0-bbe4-fdc16f590a82.16313306.1 2019-09-05 20:08:13.951 7fdc8db37580 10 cache get: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 : type miss (requested=0x16, cached=0x13) 2019-09-05 20:08:13.952 7fdc8db37580 10 cache put: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 info.flags=0x16 2019-09-05 20:08:13.952 7fdc8db37580 10 moving default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4-fdc16 f590a82.16313306.1 to cache LRU end 2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr: name=user.rgw.acl bl.length()=145 2019-09-05 20:08:13.952 7fdc8db37580 10 updating xattr: name=user.rgw.lc bl.length()=467 2019-09-05 20:08:13.952 7fdc8db37580 10 cache get: name=default.rgw.data.root++.bucket.meta.BUCKETNAME:48efb8c3-693c-4fe0-bbe4- fdc16f590a82.16313306.1 : hit (requested=0x11, cached=0x17) 2019-09-05 20:08:13.952 7fdc8db37580 10 cls_bucket_list_ordered BUCKETNAME[48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3856921.7] start _multipart_[] num_entries 1001 2019-09-05 20:08:13.960 7fdc8db37580 2 removed watcher, disabling cache Any recomendations for forcé recreate bucket index? Regards Manuel

1 1

Heavily-linked lists.ceph.com pipermail archive now appears to lead to 404s
by Florian Haas 05 Sep '19

05 Sep '19

Hi, is there any chance the list admins could copy the pipermail archive from lists.ceph.com over to lists.ceph.io? It seems to contain an awful lot of messages referred elsewhere by their archive URL, many (all?) of which appear to now lead to 404s. Example: google "Set existing pools to use hdd device class only". The top hit is a link to http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/029078.html: $ curl -IL http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/029078.html HTTP/1.1 301 Moved Permanently Server: nginx/1.10.3 (Ubuntu) Date: Thu, 29 Aug 2019 12:48:13 GMT Content-Type: text/html Content-Length: 194 Connection: keep-alive Location: https://lists.ceph.io/pipermail/ceph-users-ceph.com/2018-August/029078.html Strict-Transport-Security: max-age=31536000 HTTP/1.1 404 Not Found Server: nginx Date: Thu, 29 Aug 2019 12:48:14 GMT Content-Type: text/html; charset=utf-8 Content-Length: 3774 Connection: keep-alive X-Frame-Options: SAMEORIGIN Vary: Accept-Language, Cookie Content-Language: en Or maybe this is just a redirect rule that needs to be cleverer or more specific, rather than the apparent catch-all .com/.io redirect? Cheers, Florian

5 7

disk failure
by solarflow99 05 Sep '19

05 Sep '19

One of the things i've come to notice is when HDD drives fail, they often recover in a short time and get added back to the cluster. This causes the data to rebalance back and forth, and if I set the noout flag I get a health warning. Is there a better way to avoid this?

4 6

Re: CephFS+NFS For VMWare
by Maged Mokhtar 05 Sep '19

05 Sep '19

this is an old thread, but could be useful for others, i found out the discrepancy in VMware vmotion speed under iSCSI is probably due the "emulate_3pc" config attribute for the LIO target. if set to 0, then yes VMWare will issue io in 64KB blocks, so the bandwidth will indeed be around 25 MB/s. If emulate_3pc is set to 1, this will trigger VMWare to use vaai extended copy, which activates LIO's xcopy functionality which uses 512KB block sizes by default. We also bumped the xcopy block size to 4M (rbd object size) which gives around 400 MB/s vmotion speed, the same speed can also be achieved via Veeam backups. /Maged On 02/07/2018 14:36, Maged Mokhtar wrote: > > Hi Nick, > > With iSCSI we reach over 150 MB/s vmotion for single vm, 1 GB/s for > 7-8 vm migrations. Since these are 64KB block sizes, latency/iops is a > large factor, you need either controllers with write back cache or all > flash . hdds without write cache will suffer even with external wal/db > on ssds, giving around 80 MB/s vmotion migration. Potentially it may > be possible to get higher vmotion speeds by using fancy striping but i > would not recommend this unless your total queue depths in all your > vms is small compared to the number of osds. > > Regarding thin provisioning, a vmdk provisioned as lazy zeroed does > have an "initial" large impact on random write performance, could be > up to 10x slower. If you are writing a random 64KB to an un-allocated > vmfs block, vmfs will first write 1MB to fill the block with zeros > then write the 64KB client data, so although a lot of data is being > written the perceived client bandwidth is very low. The performance > will gradually get better with time until the disk is fully > provisioned. It is also possible to thick eager zero the vmdk disk at > creation time. Again this is more apparent with random writes rather > than sequential or vmotion load. > > Maged > > On 2018-06-29 18:48, Nick Fisk wrote: > >> This is for us peeps using Ceph with VMWare. >> >> My current favoured solution for consuming Ceph in VMWare is via >> RBD’s formatted with XFS and exported via NFS to ESXi. This seems to >> perform better than iSCSI+VMFS which seems to not play nicely with >> Ceph’s PG contention issues particularly if working with thin >> provisioned VMDK’s. >> >> I’ve still been noticing some performance issues however, mainly >> noticeable when doing any form of storage migrations. This is largely >> due to the way vSphere transfers VM’s in 64KB IO’s at a QD of 32. >> vSphere does this so Arrays with QOS can balance the IO easier than >> if larger IO’s were submitted. However Ceph’s PG locking means that >> only one or two of these IO’s can happen at a time, seriously >> lowering throughput. Typically you won’t be able to push more than >> 20-25MB/s during a storage migration >> >> There is also another issue in that the IO needed for the XFS journal >> on the RBD, can cause contention and effectively also means every NFS >> write IO sends 2 down to Ceph. This can have an impact on latency as >> well. Due to possible PG contention caused by the XFS journal updates >> when multiple IO’s are in flight, you normally end up making more and >> more RBD’s to try and spread the load. This normally means you end up >> having to do storage migrations…..you can see where I’m getting at here. >> >> I’ve been thinking for a while that CephFS works around a lot of >> these limitations. >> >> 1.It supports fancy striping, so should mean there is less per object >> contention >> >> 2.There is no FS in the middle to maintain a journal and other >> associated IO >> >> 3.A single large NFS mount should have none of the disadvantages seen >> with a single RBD >> >> 4.No need to migrate VM’s about because of #3 >> >> 5.No need to fstrim after deleting VM’s >> >> 6.Potential to do away with pacemaker and use LVS to do active/active >> NFS as ESXi does its own locking with files >> >> With this in mind I exported a CephFS mount via NFS and then mounted >> it to an ESXi host as a test. >> >> Initial results are looking very good. I’m seeing storage migrations >> to the NFS mount going at over 200MB/s, which equates to several >> thousand IO’s and seems to be writing at the intended QD32. >> >> I need to do more testing to make sure everything works as intended, >> but like I say, promising initial results. >> >> Further testing needs to be done to see what sort of MDS performance >> is required, I would imagine that since we are mainly dealing with >> large files, it might not be that critical. I also need to consider >> the stability of CephFS, RBD is relatively simple and is in use by a >> large proportion of the Ceph community. CephFS is a lot easier to >> “upset”. >> >> Nick >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users(a)lists.ceph.com <mailto:ceph-users@lists.ceph.com> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >

1 0

Best osd scenario + ansible config?
by Yoann Moulin 05 Sep '19

05 Sep '19

Hello, I am deploying a new Nautilus cluster and I would like to know what would be the best OSD's scenario config in this case : 10x 6TB Disk OSDs (data) 2x 480G SSD previously used for journal and can be used for WAL and/or DB Is it better to put all WAL on one SSD and all DBs on the other one? Or put WAL and DB of the first 5 OSDs on the first SSD and the 5 others on the second one. A more general question, what is the impact on an OSD if we lose the WAL? The DB? Both? I plan to use EC 7+5 on 12 servers and I am OK if I lose one server temporarily. I have spare servers and I can easily add another one in this cluster. To deploy this cluster, I use ceph-ansible (stable-4.0). I am not sure how to configure the playbook to use SSD and disks with LVM. https://github.com/ceph/ceph-ansible/blob/master/docs/source/osds/scenarios… Is this good? osd_objectstore: bluestore lvm_volumes: - data: data-lv1 data_vg: data-vg1 db: db-lv1 db_vg: db-vg1 wal: wal-lv1 wal_vg: wal-vg1 - data: data-lv2 data_vg: data-vg2 db: db-lv2 db_vg: db-vg2 wal: wal-lv2 wal_vg: wal-vg2 Is it possible to let the playbook configure LVM for each disk in a mixed case? It looks like I must configure LVM before running the playbook but I am not sure if I missed something. Is wal_vg and db_vg can be identical (on VG per SSD shared with multiple OSDs)? Thanks for your help. Best regards, -- Yoann Moulin EPFL IC-IT

5 12