- ceph-users - lists.ceph.io

Possible bug with rbd export/import?
by Matt Dunavant 13 Mar '20

13 Mar '20

Hello, I think I've been running into an rbd export/import bug and wanted to see if anybody else had any experience. We're using rbd images for VM drives both with and without custom stripe sizes. When we try to export/import the drive to another ceph cluster, the VM always comes up in a busted state it can't recover from. This happens both when doing this export/import through stdin/stdout and when using a middle machine as a temp space. I remember doing this a few times in previous versions without error, so I'm not sure if this is a regression or I'm doing something different. I'm still testing this to try and track down where the issue is but wanted to post this here to see if anybody else has any experience. Example command: rbd -c /etc/ceph/cluster1.conf export pool/testvm.boot - | rbd -c /etc/ceph/cluster2.conf import - pool/testvm.boot Current cluster is on 14.2.8 and using Ubuntu 18.04 w/ 5.3.0-40-generic. Let me know if I can provide any more details to help track this down. Thanks,

4 11

MGRs failing once per day and generally slow response times
by Janek Bevendorff 13 Mar '20

13 Mar '20

Hi, I've always had some MGR stability issues with daemons crashing at random times, but since the upgrade to 14.2.8 they regularly stop responding after some time until I restart them (which I have to do at least once a day). I noticed right after the upgrade that the prometheus module was entirely unresponsive and ceph fs status took about half a minute to return. Once all the cluster chatter had settled and the PGs had been rebalanced (auto-scale was messing with PGs after the upgarde), it became usable again, but everything's still slower than before. Prometheus takes several seconds to list metrics, ceph fs status takes about 1-2 seconds. However, after some time, MGRs stop responding and are kicked from the list of standbys. With log level 5 all they are writing to the log files is this: 2020-03-11 09:30:40.539 7f8f88984700 4 mgr[prometheus] ::ffff:xxx.xxx.xxx.xxx - - [11/Mar/2020:09:30:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.15.2" 2020-03-11 09:30:41.371 7f8f9ee62700 4 mgr send_beacon standby 2020-03-11 09:30:43.392 7f8f9ee62700 4 mgr send_beacon standby 2020-03-11 09:30:45.412 7f8f9ee62700 4 mgr send_beacon standby 2020-03-11 09:30:47.436 7f8f9ee62700 4 mgr send_beacon standby 2020-03-11 09:30:49.460 7f8f9ee62700 4 mgr send_beacon standby I have seen another email on this list complaining about slow ceph fs status, I believe this issue is connected. Besides the standard always-on modules I have enabled the prometheus, dashboard, and telemetry modules. Best Janek

4 6

Cancelled: Ceph Day Oslo May 13th
by Wido den Hollander 13 Mar '20

13 Mar '20

Hi, Due to the recent developments around the COVID-19 virus we (the organizers) have decided to cancel the Ceph Day in Oslo on May 13th. Altough it's still 8 weeks away we don't know how the situation will develop and if travel will be possible or people are willing to travel. Therefor we thought it was best to cancel the event for now and to re-schedule to a later date in 2020. We haven't picked a date yet. Once chosen we'll communicate it through the regular channels. Wido

1 0

Re: Ceph Performance of Micron 5210 SATA?
by Marc Roos 13 Mar '20

13 Mar '20

If you are asking, maybe run this? [global] ioengine=libaio invalidate=1 ramp_time=30 iodepth=1 runtime=180 time_based direct=1 filename=/dev/sdf [write-4k-seq] stonewall bs=4k rw=write #write_bw_log=sdx-4k-write-seq.results #write_iops_log=sdx-4k-write-seq.results [randwrite-4k-seq] stonewall bs=4k rw=randwrite #write_bw_log=sdx-4k-randwrite-seq.results #write_iops_log=sdx-4k-randwrite-seq.results [read-4k-seq] stonewall bs=4k rw=read #write_bw_log=sdx-4k-read-seq.results #write_iops_log=sdx-4k-read-seq.results [randread-4k-seq] stonewall bs=4k rw=randread #write_bw_log=sdx-4k-randread-seq.results #write_iops_log=sdx-4k-randread-seq.results [rw-4k-seq] stonewall bs=4k rw=rw #write_bw_log=sdx-4k-rw-seq.results #write_iops_log=sdx-4k-rw-seq.results [randrw-4k-seq] stonewall bs=4k rw=randrw #write_bw_log=sdx-4k-randrw-seq.results #write_iops_log=sdx-4k-randrw-seq.results [write-128k-seq] stonewall bs=128k rw=write #write_bw_log=sdx-128k-write-seq.results #write_iops_log=sdx-128k-write-seq.results [randwrite-128k-seq] stonewall bs=128k rw=randwrite #write_bw_log=sdx-128k-randwrite-seq.results #write_iops_log=sdx-128k-randwrite-seq.results [read-128k-seq] stonewall bs=128k rw=read #write_bw_log=sdx-128k-read-seq.results #write_iops_log=sdx-128k-read-seq.results [randread-128k-seq] stonewall bs=128k rw=randread #write_bw_log=sdx-128k-randread-seq.results #write_iops_log=sdx-128k-randread-seq.results [rw-128k-seq] stonewall bs=128k rw=rw #write_bw_log=sdx-128k-rw-seq.results #write_iops_log=sdx-128k-rw-seq.results [randrw-128k-seq] stonewall bs=128k rw=randrw #write_bw_log=sdx-128k-randrw-seq.results #write_iops_log=sdx-128k-randrw-seq.results [write-1024k-seq] stonewall bs=1024k rw=write #write_bw_log=sdx-1024k-write-seq.results #write_iops_log=sdx-1024k-write-seq.results [randwrite-1024k-seq] stonewall bs=1024k rw=randwrite #write_bw_log=sdx-1024k-randwrite-seq.results #write_iops_log=sdx-1024k-randwrite-seq.results [read-1024k-seq] stonewall bs=1024k rw=read #write_bw_log=sdx-1024k-read-seq.results #write_iops_log=sdx-1024k-read-seq.results [randread-1024k-seq] stonewall bs=1024k rw=randread #write_bw_log=sdx-1024k-randread-seq.results #write_iops_log=sdx-1024k-randread-seq.results [rw-1024k-seq] stonewall bs=1024k rw=rw #write_bw_log=sdx-1024k-rw-seq.results #write_iops_log=sdx-1024k-rw-seq.results [randrw-1024k-seq] stonewall bs=1024k rw=randrw #write_bw_log=sdx-1024k-randrw-seq.results #write_iops_log=sdx-1024k-randrw-seq.results -----Original Message----- From: mj [mailto:lists@merit.unu.edu] Sent: 06 March 2020 10:01 To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Ceph Performance of Micron 5210 SATA? Last monday I performed a quick test with those two disks already, probably not that relevant, but posting it anyway: I created a two-disk ceph 'cluster' on just the one local node, and ran the following: > root@ceph:~# rados bench -p scbench 10 write --no-cleanup hints = 1 > Maintaining 16 concurrent writes of 4194304 bytes to objects of size > 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_ceph_48906 > sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) > 0 0 0 0 0 0 - 0 > 1 16 107 91 363.961 364 0.0546517 0.155121 > 2 16 206 190 379.948 396 0.1529 0.159227 > 3 16 324 308 410.614 472 0.0972163 0.151421 > 4 16 458 442 441.942 536 0.0484349 0.141799 > 5 16 590 574 459.141 528 0.0445051 0.136922 > 6 16 727 711 473.941 548 0.181066 0.134468 > 7 16 856 840 479.941 516 0.187683 0.133199 > 8 16 970 954 476.942 456 0.070753 0.132642 > 9 16 1089 1073 476.831 476 0.193608 0.133754 > 10 16 1214 1198 479.142 500 0.0999212 0.132529 > Total time run: 10.097218 > Total writes made: 1215 > Write size: 4194304 > Object size: 4194304 > Bandwidth (MB/sec): 481.321 > Stddev Bandwidth: 60.481 > Max bandwidth (MB/sec): 548 > Min bandwidth (MB/sec): 364 > Average IOPS: 120 > Stddev IOPS: 15 > Max IOPS: 137 > Min IOPS: 91 > Average Latency(s): 0.132889 > Stddev Latency(s): 0.0645579 > Max latency(s): 0.336118 > Min latency(s): 0.0117049 Do let me know what else you'd want me to do. MJ _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 4

ceph qos
by 展荣臻（信泰） 13 Mar '20

13 Mar '20

Hi everyone: There are two qos in ceph(one based on tokenbucket algorithm,another based on mclock ). Which one I can use in production environment？ Thank you

1 0

EC pool 4+2 - failed to guarantee a failure domain
by Maks Kowalik 13 Mar '20

13 Mar '20

Hello, I have created a small 16pg EC pool with k=4, m=2. Then I applied following crush rule to it: rule test_ec { id 99 type erasure min_size 5 max_size 6 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step choose indep 3 type host step chooseleaf indep 2 type osd step emit } The OSD tree looks as following: -1 43.38448 root default -9 43.38448 region lab1 -7 43.38448 room dc1.lab1 -5 43.38448 rack r1.dc1.lab1 -3 14.44896 host host1.r1.dc1.lab1 6 hdd 3.63689 osd.6 up 1.00000 1.00000 8 hdd 3.63689 osd.8 up 1.00000 1.00000 7 hdd 3.63689 osd.7 up 1.00000 1.00000 11 hdd 3.53830 osd.11 up 1.00000 1.00000 -11 14.44896 host host2.r1.dc1.lab1 4 hdd 3.63689 osd.4 up 1.00000 1.00000 9 hdd 3.63689 osd.9 up 1.00000 1.00000 5 hdd 3.63689 osd.5 up 1.00000 1.00000 10 hdd 3.53830 osd.10 up 1.00000 1.00000 -13 14.48656 host host3.r1.dc1.lab1 0 hdd 3.57590 osd.0 up 1.00000 1.00000 1 hdd 3.63689 osd.1 up 1.00000 1.00000 2 hdd 3.63689 osd.2 up 1.00000 1.00000 3 hdd 3.63689 osd.3 up 1.00000 1.00000 My expectation was that each host will contain 2 shards of any PG of the pool. When I dumped PGs, it was true, but one group is placed on OSDs 0,2,3 which will cause downtime in case of host3 failure. root@host1:~/mkw # ceph pg dump|grep "^66\."|awk '{print $17}' dumped all [4,5,7,6,1,2] [8,11,9,3,0,2] <<< - this one is problematic [6,7,10,9,2,0] [2,3,7,6,5,9] [7,8,10,5,3,1] [4,5,8,6,0,2] [7,11,9,4,1,2] [5,9,0,2,7,11] [9,5,3,1,7,8] [8,11,2,0,5,9] [2,0,8,6,10,9] [3,2,5,9,7,11] [6,7,9,5,1,2] [10,5,1,3,11,8] [4,5,7,8,2,0] [7,8,3,2,9,10] Is there a way to ensure that host failure is not disruptive to the cluster? During the experiment I used info from this thread: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-October/030227.html Kind regards, Maks Kowalik

2 1

Ceph storage distribution between pools
by alexander.v.litvak＠gmail.com 13 Mar '20

13 Mar '20

I have a small cluster with a single crash map. I use 3 pools one (Openebula VMs on rbd), cephfs_data and cephfs_metadata for cephfs. Here is my ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 94 TiB 78 TiB 17 TiB 17 TiB 17.75 TOTAL 94 TiB 78 TiB 17 TiB 17 TiB 17.75 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 3.3 TiB 6.62M 10 TiB 12.36 24 TiB cephfs_metadata 2 2.1 GiB 447.63k 2.5 GiB 0 24 TiB one 5 2.2 TiB 598.12k 6.6 TiB 8.42 24 TiB What confuses me is an even distribution of MAX_AVAIL storage between those pools. When I mount cephfs on a client host, df -h shows me pool utilization. 28T 3.4T 24T 13% I also have an old hammer cluster where I see a similar picture for ceph df for a one crash map (covering rbd, cephfs-data, cephfs-metadata) GLOBAL: SIZE AVAIL RAW USED %RAW USED 87053G 31306G 55747G 64.04 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 12907G 69.37 5700G 3312474 cephfs-data 12 2873G 33.52 5700G 5859947 cephfs-meta 13 90035k 0 5700G 443961 cloud12g 14 2857G 43.41 3726G 623737 However, df -h on clients show total cluster utilization 86T 55T 31T 65% It seems that the hammer dynamically changes allocation data between the pools on the same crush map as needed. Does nautilus do the same? In this case, does 24 TB means actually avaialble space divided 3 (all my pools are set with 3/2 replication)? Thank you and sorry for the confusion

1 0

preventing the spreading of corona virus on ceph.io
by Marc Roos 12 Mar '20

12 Mar '20

Since the spreading of the corona virus is taking such drastic proportions that flights between Europe and the US are being halted. I would suggest we show some support and we temporary use in the mailing list only :) and not :D

1 0

centos7 / nautilus where to get kernel 5.5 from?
by Marc Roos 12 Mar '20

12 Mar '20

I have default centos7 setup with nautilus. I have been asked to install 5.5 to check a 'bug'. Where should I get this from? I read that the elrepo kernel is not compiled like rhel.

2 2

Re: Single machine / multiple monitors
by Brian Topping 12 Mar '20

12 Mar '20

Ok, I think that answers my question then, thanks! Too risky to be playing with patterns that will get increasingly difficult to support over time. > On Mar 12, 2020, at 12:48 PM, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > They won’t be AFAIK. Few people ever did this. > >> On Mar 12, 2020, at 11:08 AM, Brian Topping <brian.topping(a)gmail.com> wrote: >> >> If the ceph roadmap is getting rid of named clusters, how will multiple clusters be supported? How (for instance) would `/var/lib/ceph/mon/{name}` directories be resolved?

1 0