- ceph-users - lists.ceph.io

rbd-nbd multi queue
by Bobby 15 Sep '20

15 Sep '20

Hi, I have come across an old thread (2017) on the topic rbd-nbd performance. Here: https://www.spinics.net/lists/ceph-devel/msg36645.html It says they have tried adding multi-connections support on rbd-nbd with newest nbd driver, so that the nbd driver can create multiple io queues, and each io queue is associated with one socket connection to talk to rbd-nbd for request sending and response receiving. I have to work on the simular use-case. My question is , does the current rbd-ndb tool has the multi-connections support by default so that I can create multiple IO queues in the nbd driver? Thanks Bobby

1 0

Syncing cephfs from Ceph to Ceph
by Simon Sutter 15 Sep '20

15 Sep '20

Hello, Is it possible to somehow sync a ceph from one site to a ceph form another site? I'm just using the cephfs feature and no block devices. Being able to sync cephfs pools between two sites would be great for a hot backup, in case one site fails. Thanks in advance, Simon

4 5

Unable to start mds when creating cephfs volume with erasure encoding data pool
by tri＠postix.net 15 Sep '20

15 Sep '20

Hi all, I'm using ceph Octopus version and deployed it using cephadm. The ceph documentation provides 2 ways for creating a new cephfs volume: 1. via "ceph fs volume create ..." - I can use this and it works fine with the MDS automatically deployed but there is no provision for using EC with the data pool 2. via "ceph fs new ..." - I can create the new fs volume but the MDS wasn't automatically launched. If I launch them manually using "ceph orch daemon add mds ...", the mds got terminated immediated as orphan so the new fs volume isn't usable. I'd very much appreciate any pointer to how to properly create a new cephfs volume with an erasure encoding data pool. Best regards, --THoang

3 2

Re: benchmark Ceph
by rainning 15 Sep '20

15 Sep '20

What is your Ceph version? From the test results you posted, your environment's performance is okay in regard of your setup. But there are definitely many things that can be tuned to get you better number. I normally use top, iostat, pidstat, vmstat, dstat, iperf3, blktrace, netmon, ceph admin socket to monitor system stat. ------------------ Original ------------------ From:  "Tony Liu";<tonyliu0592(a)hotmail.com>; Date:  Sep 15, 2020 To:  "rainning"<tweetypie(a)qq.com>; "ceph-users"<ceph-users(a)ceph.io>; Subject:  [ceph-users] Re: benchmark Ceph Here is the test inside VM. ================ # fio --name=test --ioengine=libaio --numjobs=1 --runtime=30 \                   --direct=1 --size=2G --end_fsync=1 \                   --rw=read --bs=4K --iodepth=1 test: (groupid=0, jobs=1): err= 0: pid=14615: Mon Sep 14 21:50:55 2020    read: IOPS=3209, BW=12.5MiB/s (13.1MB/s)(376MiB/30001msec)     slat (usec): min=3, max=162, avg= 6.91, stdev= 4.74     clat (usec): min=85, max=17366, avg=303.17, stdev=639.42      lat (usec): min=161, max=17373, avg=310.38, stdev=639.93     clat percentiles (usec):      |  1.00th=[  167],  5.00th=[  172], 10.00th=[  176], 20.00th=[  182],      | 30.00th=[  188], 40.00th=[  194], 50.00th=[  204], 60.00th=[  221],      | 70.00th=[  239], 80.00th=[  277], 90.00th=[  359], 95.00th=[  461],      | 99.00th=[ 3130], 99.50th=[ 5735], 99.90th=[ 8094], 99.95th=[11338],      | 99.99th=[14091]    bw (  KiB/s): min= 9688, max=15120, per=99.87%, avg=12820.51, stdev=1001.88, samples=59    iops        : min= 2422, max= 3780, avg=3205.12, stdev=250.47, samples=59   lat (usec)   : 100=0.01%, 250=74.99%, 500=20.76%, 750=2.21%, 1000=0.50%   lat (msec)   : 2=0.39%, 4=0.27%, 10=0.81%, 20=0.06%   cpu          : usr=0.65%, sys=3.06%, ctx=96287, majf=0, minf=13   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      issued rwts: total=96287,0,0,0 short=0,0,0,0 dropped=0,0,0,0      latency   : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs):    READ: bw=12.5MiB/s (13.1MB/s), 12.5MiB/s-12.5MiB/s (13.1MB/s-13.1MB/s), io=376MiB (394MB), run=30001-30001msec Disk stats (read/write):   vda: ios=95957/2, merge=0/0, ticks=29225/12, in_queue=6027, util=82.52% ================ ================ # fio --name=test --ioengine=libaio --numjobs=1 --runtime=30 \                   --direct=1 --size=2G --end_fsync=1 \                   --rw=write --bs=4K --iodepth=1 test: (groupid=0, jobs=1): err= 0: pid=14619: Mon Sep 14 21:52:04 2020   write: IOPS=16.3k, BW=63.7MiB/s (66.8MB/s)(1917MiB/30074msec)     slat (usec): min=3, max=182, avg= 5.94, stdev= 1.30     clat (usec): min=11, max=5234, avg=54.08, stdev=18.58      lat (usec): min=35, max=5254, avg=60.26, stdev=18.80     clat percentiles (usec):      |  1.00th=[   36],  5.00th=[   38], 10.00th=[   40], 20.00th=[   46],      | 30.00th=[   48], 40.00th=[   50], 50.00th=[   53], 60.00th=[   56],      | 70.00th=[   59], 80.00th=[   63], 90.00th=[   67], 95.00th=[   71],      | 99.00th=[   85], 99.50th=[  100], 99.90th=[  289], 99.95th=[  355],      | 99.99th=[  412]    bw (  KiB/s): min=59640, max=80982, per=100.00%, avg=65462.25, stdev=7166.81, samples=59    iops        : min=14910, max=20245, avg=16365.54, stdev=1791.69, samples=59   lat (usec)   : 20=0.01%, 50=39.85%, 100=59.65%, 250=0.36%, 500=0.14%   lat (usec)   : 750=0.01%, 1000=0.01%   lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%   cpu          : usr=2.10%, sys=11.63%, ctx=490639, majf=0, minf=12   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      issued rwts: total=0,490635,0,1 short=0,0,0,0 dropped=0,0,0,0      latency   : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs):   WRITE: bw=63.7MiB/s (66.8MB/s), 63.7MiB/s-63.7MiB/s (66.8MB/s-66.8MB/s), io=1917MiB (2010MB), run=30074-30074msec Disk stats (read/write):   vda: ios=9/490639, merge=0/0, ticks=26/27102, in_queue=184, util=99.36% ================ Both networking and storage workloads are light. Which system stat should I monitor? Thanks! Tony > -----Original Message----- > From: rainning <tweetypie(a)qq.com> > Sent: Monday, September 14, 2020 8:39 PM > To: Tony Liu <tonyliu0592(a)hotmail.com>; ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: benchmark Ceph > > Can you post the fio results with the ioengine using libaio? From what > you posted, it seems to me that the read test hit cache. And the write > performance was not good, the latency was too high (~35.4ms) while the > numjobs and iodepth both were 1. Did you monitor system stat on both > side (VM/Compute Node and Cluster)? > > > > > ------------------&nbsp;Original&nbsp;------------------ > From: &nbsp;"Tony Liu";<tonyliu0592(a)hotmail.com&gt;; > Date: &nbsp;Sep 15, 2020 > To: &nbsp;"ceph-users"<ceph-users(a)ceph.io&gt;; > > Subject: &nbsp;[ceph-users] benchmark Ceph > > > > Hi, > > I have a 3-OSD-node Ceph cluster with 1 480GB SSD and 8 x 2TB 12Gpbs SAS > HDD on each node, to provide storage to a OpenStack cluster. Both public > and cluster networks are 2x10G. WAL and DB of each OSD is on SSD and > they share the same 60GB partition. > > I run fio with different combinations of operation, block size and io- > depth to collect IOPS, bandwidth and latency. I tried fio on compute > node with ioengine=rbd, also fio within VM (backed by Ceph) with > ioengine=libaio. > > The result doesn't seem good. Here are couple examples. > ==================================== > fio --name=test --ioengine=rbd --clientname=admin \ &nbsp;&nbsp;&nbsp; - > -pool=benchmark --rbdname=test --numjobs=1 \ &nbsp;&nbsp;&nbsp; -- > runtime=30 --direct=1 --size=2G \ &nbsp;&nbsp;&nbsp; --rw=read --bs=4k - > -iodepth=1 > > test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B- > 4096B, ioengine=rbd, iodepth=1 > fio-3.7 > Starting 1 process > Jobs: 1 (f=0): [f(1)][100.0%][r=27.6MiB/s,w=0KiB/s][r=7075,w=0 IOPS][eta > 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=56310: Mon Sep 14 19:01:24 2020 > &nbsp;&nbsp; read: IOPS=7610, BW=29.7MiB/s (31.2MB/s)(892MiB/30001msec) > &nbsp;&nbsp;&nbsp; slat (nsec): min=1550, max=57662, avg=3312.74, > stdev=2981.42 &nbsp;&nbsp;&nbsp; clat (usec): min=77, max=4799, > avg=127.39, stdev=39.88 &nbsp;&nbsp;&nbsp;&nbsp; lat (usec): min=78, > max=4812, avg=130.70, stdev=40.67 &nbsp;&nbsp;&nbsp; clat percentiles > (usec): > &nbsp;&nbsp;&nbsp;&nbsp; |&nbsp; 1.00th=[&nbsp;&nbsp; 82],&nbsp; > 5.00th=[&nbsp;&nbsp; 86], 10.00th=[&nbsp;&nbsp; 95], > 20.00th=[&nbsp;&nbsp; 98], &nbsp;&nbsp;&nbsp;&nbsp; | 30.00th=[&nbsp; > 100], 40.00th=[&nbsp; 104], 50.00th=[&nbsp; 116], 60.00th=[&nbsp; 129], > &nbsp;&nbsp;&nbsp;&nbsp; | 70.00th=[&nbsp; 141], 80.00th=[&nbsp; 157], > 90.00th=[&nbsp; 182], 95.00th=[&nbsp; 198], &nbsp;&nbsp;&nbsp;&nbsp; | > 99.00th=[&nbsp; 233], 99.50th=[&nbsp; 245], 99.90th=[&nbsp; 359], > 99.95th=[&nbsp; 515], &nbsp;&nbsp;&nbsp;&nbsp; | 99.99th=[&nbsp; 709] > &nbsp;&nbsp; bw (&nbsp; KiB/s): min=27160, max=40696, per=100.00%, > avg=30474.29, stdev=2826.23, samples=59 &nbsp;&nbsp; > iops&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : min= 6790, max=10174, > avg=7618.56, stdev=706.56, samples=59 &nbsp; lat (usec)&nbsp;&nbsp; : > 100=28.89%, 250=70.72%, 500=0.34%, 750=0.05%, 1000=0.01% &nbsp; lat > (msec)&nbsp;&nbsp; : 2=0.01%, 10=0.01% &nbsp; > cpu&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : usr=3.55%, > sys=3.80%, ctx=228358, majf=0, minf=29 &nbsp; IO > depths&nbsp;&nbsp;&nbsp; : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, > 32=0.0%, &gt;=64=0.0% &nbsp;&nbsp;&nbsp;&nbsp; submit&nbsp;&nbsp;&nbsp; : > 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% > &nbsp;&nbsp;&nbsp;&nbsp; complete&nbsp; : 0=0.0%, 4=100.0%, 8=0.0%, > 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% &nbsp;&nbsp;&nbsp;&nbsp; issued > rwts: total=228333,0,0,0 short=0,0,0,0 dropped=0,0,0,0 > &nbsp;&nbsp;&nbsp;&nbsp; latency&nbsp;&nbsp; : target=0, window=0, > percentile=100.00%, depth=1 > > Run status group 0 (all jobs): > &nbsp;&nbsp; READ: bw=29.7MiB/s (31.2MB/s), 29.7MiB/s-29.7MiB/s > (31.2MB/s-31.2MB/s), io=892MiB (935MB), run=30001-30001msec > > Disk stats (read/write): > &nbsp;&nbsp;&nbsp; dm-0: ios=290/3, merge=0/0, ticks=2427/19, > in_queue=2446, util=0.95%, aggrios=290/4, aggrmerge=0/0, > aggrticks=2427/39, aggrin_queue=2332, aggrutil=0.95% &nbsp; sda: > ios=290/4, merge=0/0, ticks=2427/39, in_queue=2332, util=0.95% > ==================================== > ==================================== > fio --name=test --ioengine=rbd --clientname=admin \ &nbsp;&nbsp;&nbsp; - > -pool=benchmark --rbdname=test --numjobs=1 \ &nbsp;&nbsp;&nbsp; -- > runtime=30 --direct=1 --size=2G \ &nbsp;&nbsp;&nbsp; --rw=write --bs=4k > --iodepth=1 > > test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B- > 4096B, ioengine=rbd, iodepth=1 > fio-3.7 > Starting 1 process > Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=6352KiB/s][r=0,w=1588 IOPS][eta > 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=56544: Mon Sep 14 19:03:36 2020 > &nbsp; write: IOPS=1604, BW=6417KiB/s (6571kB/s)(188MiB/30003msec) > &nbsp;&nbsp;&nbsp; slat (nsec): min=2240, max=45925, avg=6526.95, > stdev=3486.19 &nbsp;&nbsp;&nbsp; clat (usec): min=399, max=35411, > avg=615.88, stdev=231.41 &nbsp;&nbsp;&nbsp;&nbsp; lat (usec): min=402, > max=35421, avg=622.40, stdev=232.08 &nbsp;&nbsp;&nbsp; clat percentiles > (usec): > &nbsp;&nbsp;&nbsp;&nbsp; |&nbsp; 1.00th=[&nbsp; 420],&nbsp; > 5.00th=[&nbsp; 449], 10.00th=[&nbsp; 469], 20.00th=[&nbsp; 498], > &nbsp;&nbsp;&nbsp;&nbsp; | 30.00th=[&nbsp; 529], 40.00th=[&nbsp; 562], > 50.00th=[&nbsp; 611], 60.00th=[&nbsp; 652], &nbsp;&nbsp;&nbsp;&nbsp; | > 70.00th=[&nbsp; 685], 80.00th=[&nbsp; 709], 90.00th=[&nbsp; 766], > 95.00th=[&nbsp; 799], &nbsp;&nbsp;&nbsp;&nbsp; | 99.00th=[&nbsp; 881], > 99.50th=[&nbsp; 955], 99.90th=[ 2671], 99.95th=[ 3097], > &nbsp;&nbsp;&nbsp;&nbsp; | 99.99th=[ 3785] &nbsp;&nbsp; bw (&nbsp; > KiB/s): min= 5944, max= 6792, per=100.00%, avg=6415.95, stdev=178.72, > samples=60 &nbsp;&nbsp; iops&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : > min= 1486, max= 1698, avg=1603.93, stdev=44.67, samples=60 &nbsp; lat > (usec)&nbsp;&nbsp; : 500=20.82%, 750=67.23%, 1000=11.55% &nbsp; lat > (msec)&nbsp;&nbsp; : 2=0.25%, 4=0.14%, 10=0.01%, 20=0.01%, 50=0.01% > &nbsp; cpu&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : > usr=1.22%, sys=1.25%, ctx=48143, majf=0, minf=18 &nbsp; IO > depths&nbsp;&nbsp;&nbsp; : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, > 32=0.0%, &gt;=64=0.0% &nbsp;&nbsp;&nbsp;&nbsp; submit&nbsp;&nbsp;&nbsp; : > 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% > &nbsp;&nbsp;&nbsp;&nbsp; complete&nbsp; : 0=0.0%, 4=100.0%, 8=0.0%, > 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% &nbsp;&nbsp;&nbsp;&nbsp; issued > rwts: total=0,48129,0,0 short=0,0,0,0 dropped=0,0,0,0 > &nbsp;&nbsp;&nbsp;&nbsp; latency&nbsp;&nbsp; : target=0, window=0, > percentile=100.00%, depth=1 > > Run status group 0 (all jobs): > &nbsp; WRITE: bw=6417KiB/s (6571kB/s), 6417KiB/s-6417KiB/s (6571kB/s- > 6571kB/s), io=188MiB (197MB), run=30003-30003msec > > Disk stats (read/write): > &nbsp;&nbsp;&nbsp; dm-0: ios=31/2, merge=0/0, ticks=342/14, in_queue=356, > util=0.12%, aggrios=33/3, aggrmerge=0/0, aggrticks=390/27, > aggrin_queue=404, aggrutil=0.13% &nbsp; sda: ios=33/3, merge=0/0, > ticks=390/27, in_queue=404, util=0.13% > ==================================== > > Does that make sense? How do you benchmark your Ceph cluster? > Appreciate if you could share your experiences here. > > Thanks! > Tony > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Nautilus: rbd image stuck unaccessible after VM restart
by islepnev＠gmail.com 15 Sep '20

15 Sep '20

Hello, I’m running kvm virtualization with rbd storage, some images on rbd pool become efficiently unusable after VM restart. All I/O to problematic rbd image blocks infinitely. Checked that it is not a permission or locking problem. The bug was silent until we performed a planned restart of few VMs and some of VMs failed to start (kvm process timed out). It could be related to recent upgrades luminous to nautilus or proxmox 5 to 6. Ceph backend is clean, no observable problems, all mons/mgrs/osds up and running. Network is ok. Nothing in logs relevant to the problem. ceph version 14.2.6 (ba51347bdbe28c7c0e2e9172fa2983111137bb60) nautilus (stable) kernel 5.3.13-2-pve #1 SMP PVE 5.3.13-2 (Fri, 24 Jan 2020 09:49:36 +0100) x86_64 GNU/Linux HEALTH_OK No locks: # rbd status rbd-technet/vm-402-disk-0 Watchers: none # rbd status rbd-technet/vm-402-disk-1 Watchers: none Normal image vs problematic: # rbd object-map check rbd-technet/vm-402-disk-0 Object Map Check: 100% complete…done. # rbd object-map check rbd-technet/vm-402-disk-1 ^C disk-0 is good while disk-1 is effectively lost. Command hangs for many minutes with no visible activity, interrupted. rbd export runs without problems, however some data is lost after being imported back (ext4 errors). rbd deep copy worked for me. Copy looks good, no errors. # rbd info rbd-technet/vm-402-disk-1 rbd image 'vm-402-disk-1': size 16 GiB in 4096 objects order 22 (4 MiB objects) snapshot_count: 0 id: c600d06b8b4567 block_name_prefix: rbd_data.c600d06b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Fri Jan 31 17:50:50 2020 access_timestamp: Sat Mar 7 00:30:53 2020 modify_timestamp: Sat Mar 7 00:33:35 2020 journal: c600d06b8b4567 mirroring state: disabled What can be done to debug this problem? Thanks, Ilia.

4 3

Re: benchmark Ceph
by rainning 15 Sep '20

15 Sep '20

Can you post the fio results with the ioengine using libaio? From what you posted, it seems to me that the read test hit cache. And the write performance was not good, the latency was too high (~35.4ms) while the numjobs and iodepth both were 1. Did you monitor system stat on both side (VM/Compute Node and Cluster)? ------------------ Original ------------------ From:  "Tony Liu";<tonyliu0592(a)hotmail.com>; Date:  Sep 15, 2020 To:  "ceph-users"<ceph-users(a)ceph.io>; Subject:  [ceph-users] benchmark Ceph Hi, I have a 3-OSD-node Ceph cluster with 1 480GB SSD and 8 x 2TB 12Gpbs SAS HDD on each node, to provide storage to a OpenStack cluster. Both public and cluster networks are 2x10G. WAL and DB of each OSD is on SSD and they share the same 60GB partition. I run fio with different combinations of operation, block size and io-depth to collect IOPS, bandwidth and latency. I tried fio on compute node with ioengine=rbd, also fio within VM (backed by Ceph) with ioengine=libaio. The result doesn't seem good. Here are couple examples. ==================================== fio --name=test --ioengine=rbd --clientname=admin \     --pool=benchmark --rbdname=test --numjobs=1 \     --runtime=30 --direct=1 --size=2G \     --rw=read --bs=4k --iodepth=1 test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.7 Starting 1 process Jobs: 1 (f=0): [f(1)][100.0%][r=27.6MiB/s,w=0KiB/s][r=7075,w=0 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=56310: Mon Sep 14 19:01:24 2020    read: IOPS=7610, BW=29.7MiB/s (31.2MB/s)(892MiB/30001msec)     slat (nsec): min=1550, max=57662, avg=3312.74, stdev=2981.42     clat (usec): min=77, max=4799, avg=127.39, stdev=39.88      lat (usec): min=78, max=4812, avg=130.70, stdev=40.67     clat percentiles (usec):      |  1.00th=[   82],  5.00th=[   86], 10.00th=[   95], 20.00th=[   98],      | 30.00th=[  100], 40.00th=[  104], 50.00th=[  116], 60.00th=[  129],      | 70.00th=[  141], 80.00th=[  157], 90.00th=[  182], 95.00th=[  198],      | 99.00th=[  233], 99.50th=[  245], 99.90th=[  359], 99.95th=[  515],      | 99.99th=[  709]    bw (  KiB/s): min=27160, max=40696, per=100.00%, avg=30474.29, stdev=2826.23, samples=59    iops        : min= 6790, max=10174, avg=7618.56, stdev=706.56, samples=59   lat (usec)   : 100=28.89%, 250=70.72%, 500=0.34%, 750=0.05%, 1000=0.01%   lat (msec)   : 2=0.01%, 10=0.01%   cpu          : usr=3.55%, sys=3.80%, ctx=228358, majf=0, minf=29   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      issued rwts: total=228333,0,0,0 short=0,0,0,0 dropped=0,0,0,0      latency   : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs):    READ: bw=29.7MiB/s (31.2MB/s), 29.7MiB/s-29.7MiB/s (31.2MB/s-31.2MB/s), io=892MiB (935MB), run=30001-30001msec Disk stats (read/write):     dm-0: ios=290/3, merge=0/0, ticks=2427/19, in_queue=2446, util=0.95%, aggrios=290/4, aggrmerge=0/0, aggrticks=2427/39, aggrin_queue=2332, aggrutil=0.95%   sda: ios=290/4, merge=0/0, ticks=2427/39, in_queue=2332, util=0.95% ==================================== ==================================== fio --name=test --ioengine=rbd --clientname=admin \     --pool=benchmark --rbdname=test --numjobs=1 \     --runtime=30 --direct=1 --size=2G \     --rw=write --bs=4k --iodepth=1 test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.7 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=6352KiB/s][r=0,w=1588 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=56544: Mon Sep 14 19:03:36 2020   write: IOPS=1604, BW=6417KiB/s (6571kB/s)(188MiB/30003msec)     slat (nsec): min=2240, max=45925, avg=6526.95, stdev=3486.19     clat (usec): min=399, max=35411, avg=615.88, stdev=231.41      lat (usec): min=402, max=35421, avg=622.40, stdev=232.08     clat percentiles (usec):      |  1.00th=[  420],  5.00th=[  449], 10.00th=[  469], 20.00th=[  498],      | 30.00th=[  529], 40.00th=[  562], 50.00th=[  611], 60.00th=[  652],      | 70.00th=[  685], 80.00th=[  709], 90.00th=[  766], 95.00th=[  799],      | 99.00th=[  881], 99.50th=[  955], 99.90th=[ 2671], 99.95th=[ 3097],      | 99.99th=[ 3785]    bw (  KiB/s): min= 5944, max= 6792, per=100.00%, avg=6415.95, stdev=178.72, samples=60    iops        : min= 1486, max= 1698, avg=1603.93, stdev=44.67, samples=60   lat (usec)   : 500=20.82%, 750=67.23%, 1000=11.55%   lat (msec)   : 2=0.25%, 4=0.14%, 10=0.01%, 20=0.01%, 50=0.01%   cpu          : usr=1.22%, sys=1.25%, ctx=48143, majf=0, minf=18   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      issued rwts: total=0,48129,0,0 short=0,0,0,0 dropped=0,0,0,0      latency   : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs):   WRITE: bw=6417KiB/s (6571kB/s), 6417KiB/s-6417KiB/s (6571kB/s-6571kB/s), io=188MiB (197MB), run=30003-30003msec Disk stats (read/write):     dm-0: ios=31/2, merge=0/0, ticks=342/14, in_queue=356, util=0.12%, aggrios=33/3, aggrmerge=0/0, aggrticks=390/27, aggrin_queue=404, aggrutil=0.13%   sda: ios=33/3, merge=0/0, ticks=390/27, in_queue=404, util=0.13% ==================================== Does that make sense? How do you benchmark your Ceph cluster? Appreciate if you could share your experiences here. Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

benchmark Ceph
by Tony Liu 15 Sep '20

15 Sep '20

Hi, I have a 3-OSD-node Ceph cluster with 1 480GB SSD and 8 x 2TB 12Gpbs SAS HDD on each node, to provide storage to a OpenStack cluster. Both public and cluster networks are 2x10G. WAL and DB of each OSD is on SSD and they share the same 60GB partition. I run fio with different combinations of operation, block size and io-depth to collect IOPS, bandwidth and latency. I tried fio on compute node with ioengine=rbd, also fio within VM (backed by Ceph) with ioengine=libaio. The result doesn't seem good. Here are couple examples. ==================================== fio --name=test --ioengine=rbd --clientname=admin \ --pool=benchmark --rbdname=test --numjobs=1 \ --runtime=30 --direct=1 --size=2G \ --rw=read --bs=4k --iodepth=1 test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.7 Starting 1 process Jobs: 1 (f=0): [f(1)][100.0%][r=27.6MiB/s,w=0KiB/s][r=7075,w=0 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=56310: Mon Sep 14 19:01:24 2020 read: IOPS=7610, BW=29.7MiB/s (31.2MB/s)(892MiB/30001msec) slat (nsec): min=1550, max=57662, avg=3312.74, stdev=2981.42 clat (usec): min=77, max=4799, avg=127.39, stdev=39.88 lat (usec): min=78, max=4812, avg=130.70, stdev=40.67 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 86], 10.00th=[ 95], 20.00th=[ 98], | 30.00th=[ 100], 40.00th=[ 104], 50.00th=[ 116], 60.00th=[ 129], | 70.00th=[ 141], 80.00th=[ 157], 90.00th=[ 182], 95.00th=[ 198], | 99.00th=[ 233], 99.50th=[ 245], 99.90th=[ 359], 99.95th=[ 515], | 99.99th=[ 709] bw ( KiB/s): min=27160, max=40696, per=100.00%, avg=30474.29, stdev=2826.23, samples=59 iops : min= 6790, max=10174, avg=7618.56, stdev=706.56, samples=59 lat (usec) : 100=28.89%, 250=70.72%, 500=0.34%, 750=0.05%, 1000=0.01% lat (msec) : 2=0.01%, 10=0.01% cpu : usr=3.55%, sys=3.80%, ctx=228358, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=228333,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=29.7MiB/s (31.2MB/s), 29.7MiB/s-29.7MiB/s (31.2MB/s-31.2MB/s), io=892MiB (935MB), run=30001-30001msec Disk stats (read/write): dm-0: ios=290/3, merge=0/0, ticks=2427/19, in_queue=2446, util=0.95%, aggrios=290/4, aggrmerge=0/0, aggrticks=2427/39, aggrin_queue=2332, aggrutil=0.95% sda: ios=290/4, merge=0/0, ticks=2427/39, in_queue=2332, util=0.95% ==================================== ==================================== fio --name=test --ioengine=rbd --clientname=admin \ --pool=benchmark --rbdname=test --numjobs=1 \ --runtime=30 --direct=1 --size=2G \ --rw=write --bs=4k --iodepth=1 test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.7 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=6352KiB/s][r=0,w=1588 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=56544: Mon Sep 14 19:03:36 2020 write: IOPS=1604, BW=6417KiB/s (6571kB/s)(188MiB/30003msec) slat (nsec): min=2240, max=45925, avg=6526.95, stdev=3486.19 clat (usec): min=399, max=35411, avg=615.88, stdev=231.41 lat (usec): min=402, max=35421, avg=622.40, stdev=232.08 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 449], 10.00th=[ 469], 20.00th=[ 498], | 30.00th=[ 529], 40.00th=[ 562], 50.00th=[ 611], 60.00th=[ 652], | 70.00th=[ 685], 80.00th=[ 709], 90.00th=[ 766], 95.00th=[ 799], | 99.00th=[ 881], 99.50th=[ 955], 99.90th=[ 2671], 99.95th=[ 3097], | 99.99th=[ 3785] bw ( KiB/s): min= 5944, max= 6792, per=100.00%, avg=6415.95, stdev=178.72, samples=60 iops : min= 1486, max= 1698, avg=1603.93, stdev=44.67, samples=60 lat (usec) : 500=20.82%, 750=67.23%, 1000=11.55% lat (msec) : 2=0.25%, 4=0.14%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=1.22%, sys=1.25%, ctx=48143, majf=0, minf=18 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,48129,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=6417KiB/s (6571kB/s), 6417KiB/s-6417KiB/s (6571kB/s-6571kB/s), io=188MiB (197MB), run=30003-30003msec Disk stats (read/write): dm-0: ios=31/2, merge=0/0, ticks=342/14, in_queue=356, util=0.12%, aggrios=33/3, aggrmerge=0/0, aggrticks=390/27, aggrin_queue=404, aggrutil=0.13% sda: ios=33/3, merge=0/0, ticks=390/27, in_queue=404, util=0.13% ==================================== Does that make sense? How do you benchmark your Ceph cluster? Appreciate if you could share your experiences here. Thanks! Tony

1 0

Ceph RBD iSCSI compatibility
by Salsa 15 Sep '20

15 Sep '20

I just came across a Suse documentation stating that RBD features are not iSCSI compatible. Since I had 2 cases of image corruption in this scenario in 10 days I'm wondering if my setup is to blame. So question is if it is possible to provide disks to a Windows Server 2019 via iSCSI while using rbd-mirror to backup data to a second cluster? I created all images with all features enabled. Is that compatible? -- Salsa

6 12

Choosing suitable SSD for Ceph cluster
by Hermann Himmelbauer 14 Sep '20

14 Sep '20

Hi, I am running a nice ceph (proxmox 4 / debian-8 / ceph 0.94.3) cluster on 3 nodes (supermicro X8DTT-HIBQF), 2 OSD each (2TB SATA harddisks), interconnected via Infiniband 40. Problem is that the ceph performance is quite bad (approx. 30MiB/s reading, 3-4 MiB/s writing ), so I thought about plugging into each node a PCIe to NVMe/M.2 adapter and install SSD harddisks. The idea is to have a faster ceph storage and also some storage extension. The question is now which SSDs I should use. If I understand it right, not every SSD is suitable for ceph, as is denoted at the links below: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-i… or here: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark In the first link, the Samsung SSD 950 PRO 512GB NVMe is listed as a fast SSD for ceph. As the 950 is not available anymore, I ordered a Samsung 970 1TB for testing, unfortunately, the "EVO" instead of PRO. Before equipping all nodes with these SSDs, I did some tests with "fio" as recommended, e.g. like this: fio --filename=/dev/DEVICE --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test The results are as the following: ----------------------- 1) Samsung 970 EVO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=26706MB, bw=445MiB/s, iops=113945, runt= 60001msec write: io=252576KB, bw=4.1MiB/s, iops=1052, runt= 60001msec Jobs: 4: read : io=21805MB, bw=432.7MiB/s, iops=93034, runt= 60001msec write: io=422204KB, bw=6.8MiB/s, iops=1759, runt= 60002msec Jobs: 10: read : io=26921MB, bw=448MiB/s, iops=114859, runt= 60001msec write: io=435644KB, bw=7MiB/s, iops=1815, runt= 60004msec ----------------------- So the read speed is impressive, but the write speed is really bad. Therefore I ordered the Samsung 970 PRO (1TB) as it has faster NAND chips (MLC instead of TLC). The results are, however even worse for writing: ----------------------- Samsung 970 PRO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=15570MB, bw=259.4MiB/s, iops=66430, runt= 60001msec write: io=199436KB, bw=3.2MiB/s, iops=830, runt= 60001msec Jobs: 4: read : io=48982MB, bw=816.3MiB/s, iops=208986, runt= 60001msec write: io=327800KB, bw=5.3MiB/s, iops=1365, runt= 60002msec Jobs: 10: read : io=91753MB, bw=1529.3MiB/s, iops=391474, runt= 60001msec write: io=343368KB, bw=5.6MiB/s, iops=1430, runt= 60005msec ----------------------- I did some research and found out, that the "--sync" flag sets the flag "O_DSYNC" which seems to disable the SSD cache which leads to these horrid write speeds. It seems that this relates to the fact that the write cache is only not disabled for SSDs which implement some kind of battery buffer that guarantees a data flush to the flash in case of a powerloss. However, It seems impossible to find out which SSDs do have this powerloss protection, moreover, these enterprise SSDs are crazy expensive compared to the SSDs above - moreover it's unclear if powerloss protection is even available in the NVMe form factor. So building a 1 or 2 TB cluster seems not really affordable/viable. So, can please anyone give me hints what to do? Is it possible to ensure that the write cache is not disabled in some way (my server is situated in a data center, so there will probably never be loss of power). Or is the link above already outdated as newer ceph releases somehow deal with this problem? Or maybe a later Debian release (10) will handle the O_DSYNC flag differently? Perhaps I should simply invest in faster (and bigger) harddisks and forget the SSD-cluster idea? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

12 18

ceph-container: docker restart, mon's unable to join
by Stefan Kooman 14 Sep '20

14 Sep '20

Hi, In an attempt to get a (test) Mimic cluster running on Ubuntu 20.04 we are using docker with ceph-container images (ceph/daemon:latest-mimic). Deploying monitors and mgrs works fine. If however a monitor container gets stopped and started (i.e. docker restart) two out of three (with exception of mon initial member) mons won't join the cluster anymore and keep logging the following: /opt/ceph-container/bin/entrypoint.sh: Existing mon, trying to rejoin cluster... If docker is stopped, the mon directory "/var/lib/ceph/mon/$mon-name" removed and docker started again the mon is able to join the cluster. This directory is a persistent volume with correct permissions (167.167). No etcd cluster is in use here. We manually copied the /etc/ceph and /var/lib/ceph directories to the docker hosts. Any hints on how to make a mon container survive a reboot is welcome. Gr. Stefan P.s And yes, we know about Rook, kubernetes, etc. but that's not what want to use now.

1 0