- ceph-users - lists.ceph.io

Ceph BoF at SCALE 18x
by Mike Perez 13 Jan '20

13 Jan '20

Hi everyone, SCALE 18x is taking place at the same time as of Cephalocon 2020 in Seoul. I wanted to see if there is interest in me registering us a BoF slot and if anybody in the area will be attending to run it? https://www.socallinuxexpo.org/scale/18x/bird-feather-sessions -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

where does 100% RBD utilization come from?
by Philip Brown 10 Jan '20

10 Jan '20

oops. I posted this to the "Old" list, but supposedly this is the new list and the better place to ask questions? A google search didnt seem to find the answer on this, so thought I'd ask here: what determines if an rdb is "100% busy"? I have some backend OSDs, and an iSCSI gateway, serving out some RBDs. iostat on the gateway says rbd is 100% utilized iostat on individual OSds only goes as high as about 60% on a per-device basis. CPU is idle. Doesnt seem like network interface is capped either. So.. how do I improve RBD throughput?

1 0

heads up about the pg autoscaler
by Dan van der Ster 10 Jan '20

10 Jan '20

Hi all, Just a quick heads up that if you're using the pg-autoscaler with EC pools, it looks to me that it creates too many PGs. See https://tracker.ceph.com/issues/43546 for the details. As a workaround you can decrease mon_target_pg_per_osd accordingly. Cheers, Dan

1 0

Trying to install nautilus, keep getting mimic
by Jorge Garcia 10 Jan '20

10 Jan '20

I have been trying to install Nautilus using ceph-deploy, but somehow keep getting mimic installed. I'm not sure where mimic is even coming from... Here's what I'm doing (basically following the docs): * Fresh install of Centos 7.7 sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm cat << EOM > /etc/yum.repos.d/ceph.repo [ceph-noarch] name=Ceph noarch packages baseurl=https://download.ceph.com/rpm-nautilus/el7/noarch enabled=1 gpgcheck=1 type=rpm-md gpgkey=https://download.ceph.com/keys/release.asc EOM sudo yum update sudo yum install ceph-deploy * I also install ntp and openssh * Create a ceph-deploy user, enable sudo and distribute keys mkdir my-cluster cd my-cluster ceph-deploy new node1 ceph-deploy install node1 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephpub/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy install node1 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] testing : None [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f26ef6e4518> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] dev_commit : None [ceph_deploy.cli][INFO ] install_mds : False [ceph_deploy.cli][INFO ] stable : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] adjust_repos : True [ceph_deploy.cli][INFO ] func : <function install at 0x7f26f07c3a28> [ceph_deploy.cli][INFO ] install_mgr : False [ceph_deploy.cli][INFO ] install_all : False [ceph_deploy.cli][INFO ] repo : False [ceph_deploy.cli][INFO ] host : ['node1'] [ceph_deploy.cli][INFO ] install_rgw : False [ceph_deploy.cli][INFO ] install_tests : False [ceph_deploy.cli][INFO ] repo_url : None [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] install_osd : False [ceph_deploy.cli][INFO ] version_kind : stable [ceph_deploy.cli][INFO ] install_common : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] dev : master [ceph_deploy.cli][INFO ] nogpgcheck : False [ceph_deploy.cli][INFO ] local_mirror : None [ceph_deploy.cli][INFO ] release : None [ceph_deploy.cli][INFO ] install_mon : False [ceph_deploy.cli][INFO ] gpg_url : None [ceph_deploy.install][DEBUG ] Installing stable version *mimic* on cluster ceph hosts node1 Where did it get that I wanted to install *mimic*? It all goes downhill from there, with my repo getting replaced by the mimic repo and mimic getting installed. What am I missing?

2 1

Multi-site clusters
by eduard.rushanyan＠centro.net 09 Jan '20

09 Jan '20

CEPH Community! Cheers to 2020 We have a single cluster running in two modes: CephFS as mountable filesystem and Rados Object Gateway (S3 like for storing objects) - all good there no questions, CEPH is awesome! Were planning to get 2nd cluster in a different Datacenter and have both clusters connected / replicated / mirrored (e.g. multi-site). Is there a way to achieve that for CephFS and Rados modes, any ideas / recommendations / best practices / articles someone can share? Any insight would be much, much appreciated! P.S. The RBD mirroring which seemed like is the solution - went as far as discovering that RBD mirroring works for block devices only, can’t be used with CephFS. There are some references as to Rados supporting multi-site with potentially significant impact on write latency. Thanks, Eduard

1 1

v14.2.6 Nautilus released
by Abhishek Lekshmanan 09 Jan '20

09 Jan '20

We're happy to announce the availability of the sixth update to the Ceph Nautilus release series. This is a hotfix release primarily fixing a regression introduced in v14.2.5, all nautilus users are advised to upgrade to this release. Notable Changes --------------- * This release fixes a `ceph-mgr` bug that caused mgr becoming unresponsive on larger clusters issue#43364 https://tracker.ceph.com/issues/43364, pr#32466 (https://github.com/ceph/ceph/pull/32466 David Zafman, Neha Ojha) The release notes can be also found at https://ceph.io/releases/v14-2-6-nautilus-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.6.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH

1 0

S3 Object Lock feature in 14.2.5
by Robert Sander 09 Jan '20

09 Jan '20

Hi, Ceph Nautilus 14.2.5 got the S3 Object Lock feature backported. It should now be possible to create such a bucket with: s3cmd --add-header=x-amz-bucket-object-lock-enabled:true mb s3://lockedbucket But an S3 client does not recognize the bucket having the Object Lock feature. Am I missing something here? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

why osd's heartbeat partner comes from another root tree?
by opengers 08 Jan '20

08 Jan '20

Hello: According to my understanding, osd's heartbeat partners only come from those osds who assume the same pg See below(# ceph osd tree), osd.10 and osd.0-6 cannot assume the same pg, because osd.10 and osd.0-6 are from different root tree, and pg in my cluster doesn't map across root trees(# ceph osd crush rule dump). so, osd.0-6 cannot become the heartbeat partner of osd.10 But, below is the log on osd.10, It can be seen that the osd.10's heartbeat partner include osd.0/1/2/5, why? thanks for any help > # osd.10 log > 2019-11-20 09:21:50.431799 7fbb369fb700 -1 osd.10 7344 heartbeat_check: no > reply from 10.13.6.162:6806 osd.2 since back 2019-11-20 09:21:19.979712 > front 2019-11-20 09:21:19.979712 (cutoff 2019-11-20 09:21:30.431768) > 2019-11-20 13:15:59.175060 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no > reply from 10.13.6.162:6806 <http://10.13.6.152:6806> osd.2 since back > 2019-11-20 13:15:38.710424 front 2019-11-20 13:15:38.710424 (cutoff > 2019-11-20 13:15:39.175058) > 2019-11-20 13:15:59.175110 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no > reply from 10.13.6.160:6803 osd.0 since back 2019-11-20 13:15:38.710424 > front 2019-11-20 13:15:38.710424 (cutoff 2019-11-20 13:15:39.175058) > 2019-11-20 13:15:59.175118 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no > reply from 10.13.6.161:6803 osd.1 since back 2019-11-20 13:15:38.710424 > front 2019-11-20 13:15:38.710424 (cutoff 2019-11-20 13:15:39.175058) > 2019-11-21 02:52:24.656783 7fbb369fb700 -1 osd.10 7374 heartbeat_check: no > reply from 10.13.6.158:6810 osd.5 since back 2019-11-21 02:52:04.557548 > front 2019-11-21 02:52:04.557548 (cutoff 2019-11-21 02:52:04.656781) > # ceph osd tree > -17 3.29095 root ssd-storage > > -25 1.09698 rack rack-ssd-A > > -18 1.09698 host ssd-osd01 > 10 hdd 1.09698 osd.10 up 1.00000 > 1.00000 > -26 1.09698 rack rack-ssd-B > > -19 1.09698 host ssd-osd02 > 11 hdd 1.09698 osd.11 up 1.00000 > 1.00000 > -27 1.09698 rack rack-ssd-C > > -20 1.09698 host ssd-osd03 > 12 hdd 1.09698 osd.12 up 1.00000 > 1.00000 > -1 3.22256 root default > > -3 0.29300 host test-osd01 > > 0 hdd 0.29300 osd.0 up 1.00000 > 1.00000 > -5 0.29300 host test-osd02 > > 1 hdd 0.29300 osd.1 up 0.89999 > 1.00000 > -7 0.29300 host test-osd03 > > 2 hdd 0.29300 osd.2 up 0.79999 > 1.00000 > -9 0.29300 host test-osd04 > > 3 hdd 0.29300 osd.3 up 1.00000 > 1.00000 > -11 0.29300 host test-osd05 > > 4 hdd 0.29300 osd.4 up 1.00000 > 1.00000 > -13 0.29300 host test-osd06 > > 5 hdd 0.29300 osd.5 up 1.00000 > 1.00000 > -15 0.29300 host test-osd07 > > 6 hdd 0.29300 osd.6 up 1.00000 > 1.00000 # ceph osd crush rule dump > > [ > { > "rule_id": 0, > "rule_name": "replicated_rule", > "ruleset": 0, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -1, > "item_name": "default" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "host" > }, > { > "op": "emit" > } > ] > }, > { > "rule_id": 1, > "rule_name": "replicated_rule_ssd", > "ruleset": 1, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -17, > "item_name": "ssd-storage" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "rack" > }, > { > "op": "emit" > } > ] > } > ] > # some parameters > "mon_osd_down_out_interval": "600", > "mon_osd_down_out_subtree_limit": "rack", > "mds_debug_subtrees": "false", > "mon_osd_down_out_subtree_limit": "rack", > "mon_osd_reporter_subtree_level": "host",

3 3

Poor performance after (incomplete?) upgrade to Nautilus
by Georg F 08 Jan '20

08 Jan '20

Hi all, since a few weeks our Nautilus cluster was struggling with severe performance issues. When an OSD would go down, the rebalancing was really slow. Long periods with no data transfer at all (client and rebalancing!) and times with rebalancing traffic only. However, client traffic was almost stalled for the whole period until all objects were in place again (VMs were frozen). PGs were stuck in peering or inactive for long times. Sometimes we had to restart the ceph-mon in order to get the whole process running again. The issues started all of a sudden, we don't remember doing any changes to the configuration. The whole cluster has been updated from Mimic to Nautilus (14.2.3) in September while the issue occurred just a few weeks ago. Updating it to 14.2.5 did not resolve the issue back then. Looking through mailing lists I found the following message: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-July/028035.html So I ran "ceph osd require-osd-release nautilus" and all of a sudden the problems where gone! I do not recall executing that command right after the upgrade because the documentation states "Complete the upgrade by disallowing pre-Nautilus OSDs and enabling all new Nautilus-only functionality.". As by that point in time all OSDs, MONs and MGRs were successfully updated there was no reason to believe this command would be necessary. Therefore I got two questions: 1. What exactly does the command do besides preventing old OSDs from joining? 2. What could have been the issue with the cluster and how did this command fix it? If it is really that important to run the command, the docs should state this more clearly. I appreciate any insight on this topic. Thanks, Georg

1 0

ceph balancer <argument> runs for minutes or hangs
by Thomas Schneider 08 Jan '20

08 Jan '20

Hi, happy new year to you! I'm running a multinode cluster with 3 MGR nodes. The issue I'm facing now is that ceph balancer <argument> runs for minutes or, in worst case, hangs. I have documented the runtime of the following executions: root@ld3955:~# date && time ceph balancer status Mon Dec 23 10:06:12 CET 2019 { "active": true, "plans": [], "mode": "upmap" } real 1m45,045s user 0m0,315s sys 0m0,026s root@ld3955:~# date && time ceph balancer status Tue Jan 7 08:11:24 CET 2020 ^CInterrupted Traceback (most recent call last): File "/usr/bin/ceph", line 1263, in <module> retval = main() File "/usr/bin/ceph", line 1194, in main verbose) File "/usr/bin/ceph", line 619, in new_style_command ret, outbuf, outs = do_command(parsed_args, target, cmdargs, sigdict, inbuf, verbose) File "/usr/bin/ceph", line 593, in do_command return ret, '', '' UnboundLocalError: local variable 'ret' referenced before assignment real 102m44,084s user 0m2,404s sys 0m1,065s root@ld3955:~# date && time ceph balancer off Tue Jan 7 09:57:36 CET 2020 real 1m45,371s user 0m0,358s sys 0m0,013s root@ld3955:~# date && time ceph balancer on Tue Jan 7 14:57:03 CET 2020 real 0m0,452s user 0m0,284s sys 0m0,020s root@ld3955:~# date && time ceph balancer status Tue Jan 7 14:57:11 CET 2020 { "active": true, "plans": [], "mode": "upmap" } real 1m52,902s user 0m0,301s sys 0m0,042s root@ld3955:~# date && time ceph balancer off Wed Jan 8 08:49:26 CET 2020 ^CInterrupted Traceback (most recent call last): File "/usr/bin/ceph", line 1263, in <module> retval = main() File "/usr/bin/ceph", line 1194, in main verbose) File "/usr/bin/ceph", line 619, in new_style_command ret, outbuf, outs = do_command(parsed_args, target, cmdargs, sigdict, inbuf, verbose) File "/usr/bin/ceph", line 593, in do_command return ret, '', '' UnboundLocalError: local variable 'ret' referenced before assignment real 14m29,097s user 0m0,579s sys 0m0,157s In correlation with this finding I have identified that active MGR node is using +100% CPU, to be pricise 108-120%. To workaround this issue I must stop the active MRG node service and wait until another node becomes active. What's the issue with MGR service here? Should I open a bug report? Regards

1 0