September 2020 - ceph-users

ceph rbox test on passive compressed pool

by Marc Roos

I have been inserting 10790 exactly the same 64kb text message to a passive compressing enabled pool. I am still counting, but it looks like only half the objects are compressed. mail/b08c3218dbf1545ff430000052412a8e mtime 2020-09-06 16:27:39.000000, size 63580 mail/00f6043775f1545ff430000052412a8e mtime 2020-09-06 16:25:57.000000, size 525 mail/b875f40571f1545ff430000052412a8e mtime 2020-09-06 16:25:53.000000, size 63580 mail/e87c120b19f1545ff430000052412a8e mtime 2020-09-06 16:24:25.000000, size 525 I am not sure if this should be expected from passive, these docs[1] hint that passive 'compress if hinted COMPRESSIBLE'. From that I would conclude that all text messages should be compressed. A previous test with a 64kb gzip attachment seemed to not compress, although I did not look at all object sizes. on 14.2.11 [1] https://documentation.suse.com/ses/5.5/html/ses-all/ceph-pools.html#sec-cep… https://docs.ceph.com/docs/mimic/rados/operations/pools/

3 years, 8 months

3
8
0 0

Matlab Assignment Help

by brainycoordinator＠gmail.com

MATLAB is the easiest software and it is the most productive software. But students are unable to handle this software and look for <a href="https://sourceessay.com/online-assignment-help-assistance/">Online Assignment Help</a>. If you are looking for assignment help then you can hire best assignment writers. These writers are very professional and know all types of software and know how to complete the assignment in a professional way.

3 years, 8 months

1
0
0 0

How does the sign-in issue stop to Unlock Yahoo Account?

by mary smith

On the off chance that you get incapable to sign in to your Yahoo account, at that point you have to apply the right investigating steps to keep away from Unlock Yahoo Account circumstance. All things considered, on the off chance that you entered an inappropriate login certification, at that point your account will be blocked, and afterward, you have to unlock it to utilize it for additional. https://www.customercare-email.com/blog/unlock-my-yahoo-account/

3 years, 8 months

1
0
0 0

Can't do creating plant settings in Epson contraption? Call Epson Printer Support.

by mary smith

The gadget is known for dealing with an assortment of documents. Regardless, the records can break down and henceforth the need may ascend to fix the issue. Considering, you can utilize the help that is given in the specific assistance zones or you can find uphold from the Epson Printer Support. https://www.epsonprintersupportpro.net/

3 years, 8 months

1
0
0 0

rbd-nbd multi queue

by Bobby

Hi, I have come across an old thread (2017) on the topic rbd-nbd performance. Here: https://www.spinics.net/lists/ceph-devel/msg36645.html It says they have tried adding multi-connections support on rbd-nbd with newest nbd driver, so that the nbd driver can create multiple io queues, and each io queue is associated with one socket connection to talk to rbd-nbd for request sending and response receiving. I have to work on the simular use-case. My question is , does the current rbd-ndb tool has the multi-connections support by default so that I can create multiple IO queues in the nbd driver? Thanks Bobby

3 years, 8 months

1
0
0 0

Syncing cephfs from Ceph to Ceph

by Simon Sutter

Hello, Is it possible to somehow sync a ceph from one site to a ceph form another site? I'm just using the cephfs feature and no block devices. Being able to sync cephfs pools between two sites would be great for a hot backup, in case one site fails. Thanks in advance, Simon

3 years, 8 months

4
5
0 0

Unable to start mds when creating cephfs volume with erasure encoding data pool

by tri＠postix.net

Hi all, I'm using ceph Octopus version and deployed it using cephadm. The ceph documentation provides 2 ways for creating a new cephfs volume: 1. via "ceph fs volume create ..." - I can use this and it works fine with the MDS automatically deployed but there is no provision for using EC with the data pool 2. via "ceph fs new ..." - I can create the new fs volume but the MDS wasn't automatically launched. If I launch them manually using "ceph orch daemon add mds ...", the mds got terminated immediated as orphan so the new fs volume isn't usable. I'd very much appreciate any pointer to how to properly create a new cephfs volume with an erasure encoding data pool. Best regards, --THoang

3 years, 8 months

3
2
0 0

Re: benchmark Ceph

by rainning

What is your Ceph version? From the test results you posted, your environment's performance is okay in regard of your setup. But there are definitely many things that can be tuned to get you better number. I normally use top, iostat, pidstat, vmstat, dstat, iperf3, blktrace, netmon, ceph admin socket to monitor system stat. ------------------ Original ------------------ From:  "Tony Liu";<tonyliu0592(a)hotmail.com>gt;; Date:  Sep 15, 2020 To:  "rainning"<tweetypie(a)qq.com>gt;; "ceph-users"<ceph-users(a)ceph.io>gt;; Subject:  [ceph-users] Re: benchmark Ceph Here is the test inside VM. ================ # fio --name=test --ioengine=libaio --numjobs=1 --runtime=30 \                   --direct=1 --size=2G --end_fsync=1 \                   --rw=read --bs=4K --iodepth=1 test: (groupid=0, jobs=1): err= 0: pid=14615: Mon Sep 14 21:50:55 2020    read: IOPS=3209, BW=12.5MiB/s (13.1MB/s)(376MiB/30001msec)     slat (usec): min=3, max=162, avg= 6.91, stdev= 4.74     clat (usec): min=85, max=17366, avg=303.17, stdev=639.42      lat (usec): min=161, max=17373, avg=310.38, stdev=639.93     clat percentiles (usec):      |  1.00th=[  167],  5.00th=[  172], 10.00th=[  176], 20.00th=[  182],      | 30.00th=[  188], 40.00th=[  194], 50.00th=[  204], 60.00th=[  221],      | 70.00th=[  239], 80.00th=[  277], 90.00th=[  359], 95.00th=[  461],      | 99.00th=[ 3130], 99.50th=[ 5735], 99.90th=[ 8094], 99.95th=[11338],      | 99.99th=[14091]    bw (  KiB/s): min= 9688, max=15120, per=99.87%, avg=12820.51, stdev=1001.88, samples=59    iops        : min= 2422, max= 3780, avg=3205.12, stdev=250.47, samples=59   lat (usec)   : 100=0.01%, 250=74.99%, 500=20.76%, 750=2.21%, 1000=0.50%   lat (msec)   : 2=0.39%, 4=0.27%, 10=0.81%, 20=0.06%   cpu          : usr=0.65%, sys=3.06%, ctx=96287, majf=0, minf=13   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      issued rwts: total=96287,0,0,0 short=0,0,0,0 dropped=0,0,0,0      latency   : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs):    READ: bw=12.5MiB/s (13.1MB/s), 12.5MiB/s-12.5MiB/s (13.1MB/s-13.1MB/s), io=376MiB (394MB), run=30001-30001msec Disk stats (read/write):   vda: ios=95957/2, merge=0/0, ticks=29225/12, in_queue=6027, util=82.52% ================ ================ # fio --name=test --ioengine=libaio --numjobs=1 --runtime=30 \                   --direct=1 --size=2G --end_fsync=1 \                   --rw=write --bs=4K --iodepth=1 test: (groupid=0, jobs=1): err= 0: pid=14619: Mon Sep 14 21:52:04 2020   write: IOPS=16.3k, BW=63.7MiB/s (66.8MB/s)(1917MiB/30074msec)     slat (usec): min=3, max=182, avg= 5.94, stdev= 1.30     clat (usec): min=11, max=5234, avg=54.08, stdev=18.58      lat (usec): min=35, max=5254, avg=60.26, stdev=18.80     clat percentiles (usec):      |  1.00th=[   36],  5.00th=[   38], 10.00th=[   40], 20.00th=[   46],      | 30.00th=[   48], 40.00th=[   50], 50.00th=[   53], 60.00th=[   56],      | 70.00th=[   59], 80.00th=[   63], 90.00th=[   67], 95.00th=[   71],      | 99.00th=[   85], 99.50th=[  100], 99.90th=[  289], 99.95th=[  355],      | 99.99th=[  412]    bw (  KiB/s): min=59640, max=80982, per=100.00%, avg=65462.25, stdev=7166.81, samples=59    iops        : min=14910, max=20245, avg=16365.54, stdev=1791.69, samples=59   lat (usec)   : 20=0.01%, 50=39.85%, 100=59.65%, 250=0.36%, 500=0.14%   lat (usec)   : 750=0.01%, 1000=0.01%   lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%   cpu          : usr=2.10%, sys=11.63%, ctx=490639, majf=0, minf=12   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      issued rwts: total=0,490635,0,1 short=0,0,0,0 dropped=0,0,0,0      latency   : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs):   WRITE: bw=63.7MiB/s (66.8MB/s), 63.7MiB/s-63.7MiB/s (66.8MB/s-66.8MB/s), io=1917MiB (2010MB), run=30074-30074msec Disk stats (read/write):   vda: ios=9/490639, merge=0/0, ticks=26/27102, in_queue=184, util=99.36% ================ Both networking and storage workloads are light. Which system stat should I monitor? Thanks! Tony > -----Original Message----- > From: rainning <tweetypie(a)qq.com> > Sent: Monday, September 14, 2020 8:39 PM > To: Tony Liu <tonyliu0592(a)hotmail.com>gt;; ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: benchmark Ceph > > Can you post the fio results with the ioengine using libaio? From what > you posted, it seems to me that the read test hit cache. And the write > performance was not good, the latency was too high (~35.4ms) while the > numjobs and iodepth both were 1. Did you monitor system stat on both > side (VM/Compute Node and Cluster)? > > > > > ------------------&nbsp;Original&nbsp;------------------ > From: &nbsp;"Tony Liu";<tonyliu0592(a)hotmail.com&gt;amp;gt;; > Date: &nbsp;Sep 15, 2020 > To: &nbsp;"ceph-users"<ceph-users(a)ceph.io&gt;amp;gt;; > > Subject: &nbsp;[ceph-users] benchmark Ceph > > > > Hi, > > I have a 3-OSD-node Ceph cluster with 1 480GB SSD and 8 x 2TB 12Gpbs SAS > HDD on each node, to provide storage to a OpenStack cluster. Both public > and cluster networks are 2x10G. WAL and DB of each OSD is on SSD and > they share the same 60GB partition. > > I run fio with different combinations of operation, block size and io- > depth to collect IOPS, bandwidth and latency. I tried fio on compute > node with ioengine=rbd, also fio within VM (backed by Ceph) with > ioengine=libaio. > > The result doesn't seem good. Here are couple examples. > ==================================== > fio --name=test --ioengine=rbd --clientname=admin \ &nbsp;&nbsp;&nbsp; - > -pool=benchmark --rbdname=test --numjobs=1 \ &nbsp;&nbsp;&nbsp; -- > runtime=30 --direct=1 --size=2G \ &nbsp;&nbsp;&nbsp; --rw=read --bs=4k - > -iodepth=1 > > test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B- > 4096B, ioengine=rbd, iodepth=1 > fio-3.7 > Starting 1 process > Jobs: 1 (f=0): [f(1)][100.0%][r=27.6MiB/s,w=0KiB/s][r=7075,w=0 IOPS][eta > 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=56310: Mon Sep 14 19:01:24 2020 > &nbsp;&nbsp; read: IOPS=7610, BW=29.7MiB/s (31.2MB/s)(892MiB/30001msec) > &nbsp;&nbsp;&nbsp; slat (nsec): min=1550, max=57662, avg=3312.74, > stdev=2981.42 &nbsp;&nbsp;&nbsp; clat (usec): min=77, max=4799, > avg=127.39, stdev=39.88 &nbsp;&nbsp;&nbsp;&nbsp; lat (usec): min=78, > max=4812, avg=130.70, stdev=40.67 &nbsp;&nbsp;&nbsp; clat percentiles > (usec): > &nbsp;&nbsp;&nbsp;&nbsp; |&nbsp; 1.00th=[&nbsp;&nbsp; 82],&nbsp; > 5.00th=[&nbsp;&nbsp; 86], 10.00th=[&nbsp;&nbsp; 95], > 20.00th=[&nbsp;&nbsp; 98], &nbsp;&nbsp;&nbsp;&nbsp; | 30.00th=[&nbsp; > 100], 40.00th=[&nbsp; 104], 50.00th=[&nbsp; 116], 60.00th=[&nbsp; 129], > &nbsp;&nbsp;&nbsp;&nbsp; | 70.00th=[&nbsp; 141], 80.00th=[&nbsp; 157], > 90.00th=[&nbsp; 182], 95.00th=[&nbsp; 198], &nbsp;&nbsp;&nbsp;&nbsp; | > 99.00th=[&nbsp; 233], 99.50th=[&nbsp; 245], 99.90th=[&nbsp; 359], > 99.95th=[&nbsp; 515], &nbsp;&nbsp;&nbsp;&nbsp; | 99.99th=[&nbsp; 709] > &nbsp;&nbsp; bw (&nbsp; KiB/s): min=27160, max=40696, per=100.00%, > avg=30474.29, stdev=2826.23, samples=59 &nbsp;&nbsp; > iops&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : min= 6790, max=10174, > avg=7618.56, stdev=706.56, samples=59 &nbsp; lat (usec)&nbsp;&nbsp; : > 100=28.89%, 250=70.72%, 500=0.34%, 750=0.05%, 1000=0.01% &nbsp; lat > (msec)&nbsp;&nbsp; : 2=0.01%, 10=0.01% &nbsp; > cpu&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : usr=3.55%, > sys=3.80%, ctx=228358, majf=0, minf=29 &nbsp; IO > depths&nbsp;&nbsp;&nbsp; : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, > 32=0.0%, &gt;=64=0.0% &nbsp;&nbsp;&nbsp;&nbsp; submit&nbsp;&nbsp;&nbsp; : > 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% > &nbsp;&nbsp;&nbsp;&nbsp; complete&nbsp; : 0=0.0%, 4=100.0%, 8=0.0%, > 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% &nbsp;&nbsp;&nbsp;&nbsp; issued > rwts: total=228333,0,0,0 short=0,0,0,0 dropped=0,0,0,0 > &nbsp;&nbsp;&nbsp;&nbsp; latency&nbsp;&nbsp; : target=0, window=0, > percentile=100.00%, depth=1 > > Run status group 0 (all jobs): > &nbsp;&nbsp; READ: bw=29.7MiB/s (31.2MB/s), 29.7MiB/s-29.7MiB/s > (31.2MB/s-31.2MB/s), io=892MiB (935MB), run=30001-30001msec > > Disk stats (read/write): > &nbsp;&nbsp;&nbsp; dm-0: ios=290/3, merge=0/0, ticks=2427/19, > in_queue=2446, util=0.95%, aggrios=290/4, aggrmerge=0/0, > aggrticks=2427/39, aggrin_queue=2332, aggrutil=0.95% &nbsp; sda: > ios=290/4, merge=0/0, ticks=2427/39, in_queue=2332, util=0.95% > ==================================== > ==================================== > fio --name=test --ioengine=rbd --clientname=admin \ &nbsp;&nbsp;&nbsp; - > -pool=benchmark --rbdname=test --numjobs=1 \ &nbsp;&nbsp;&nbsp; -- > runtime=30 --direct=1 --size=2G \ &nbsp;&nbsp;&nbsp; --rw=write --bs=4k > --iodepth=1 > > test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B- > 4096B, ioengine=rbd, iodepth=1 > fio-3.7 > Starting 1 process > Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=6352KiB/s][r=0,w=1588 IOPS][eta > 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=56544: Mon Sep 14 19:03:36 2020 > &nbsp; write: IOPS=1604, BW=6417KiB/s (6571kB/s)(188MiB/30003msec) > &nbsp;&nbsp;&nbsp; slat (nsec): min=2240, max=45925, avg=6526.95, > stdev=3486.19 &nbsp;&nbsp;&nbsp; clat (usec): min=399, max=35411, > avg=615.88, stdev=231.41 &nbsp;&nbsp;&nbsp;&nbsp; lat (usec): min=402, > max=35421, avg=622.40, stdev=232.08 &nbsp;&nbsp;&nbsp; clat percentiles > (usec): > &nbsp;&nbsp;&nbsp;&nbsp; |&nbsp; 1.00th=[&nbsp; 420],&nbsp; > 5.00th=[&nbsp; 449], 10.00th=[&nbsp; 469], 20.00th=[&nbsp; 498], > &nbsp;&nbsp;&nbsp;&nbsp; | 30.00th=[&nbsp; 529], 40.00th=[&nbsp; 562], > 50.00th=[&nbsp; 611], 60.00th=[&nbsp; 652], &nbsp;&nbsp;&nbsp;&nbsp; | > 70.00th=[&nbsp; 685], 80.00th=[&nbsp; 709], 90.00th=[&nbsp; 766], > 95.00th=[&nbsp; 799], &nbsp;&nbsp;&nbsp;&nbsp; | 99.00th=[&nbsp; 881], > 99.50th=[&nbsp; 955], 99.90th=[ 2671], 99.95th=[ 3097], > &nbsp;&nbsp;&nbsp;&nbsp; | 99.99th=[ 3785] &nbsp;&nbsp; bw (&nbsp; > KiB/s): min= 5944, max= 6792, per=100.00%, avg=6415.95, stdev=178.72, > samples=60 &nbsp;&nbsp; iops&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : > min= 1486, max= 1698, avg=1603.93, stdev=44.67, samples=60 &nbsp; lat > (usec)&nbsp;&nbsp; : 500=20.82%, 750=67.23%, 1000=11.55% &nbsp; lat > (msec)&nbsp;&nbsp; : 2=0.25%, 4=0.14%, 10=0.01%, 20=0.01%, 50=0.01% > &nbsp; cpu&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : > usr=1.22%, sys=1.25%, ctx=48143, majf=0, minf=18 &nbsp; IO > depths&nbsp;&nbsp;&nbsp; : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, > 32=0.0%, &gt;=64=0.0% &nbsp;&nbsp;&nbsp;&nbsp; submit&nbsp;&nbsp;&nbsp; : > 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% > &nbsp;&nbsp;&nbsp;&nbsp; complete&nbsp; : 0=0.0%, 4=100.0%, 8=0.0%, > 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% &nbsp;&nbsp;&nbsp;&nbsp; issued > rwts: total=0,48129,0,0 short=0,0,0,0 dropped=0,0,0,0 > &nbsp;&nbsp;&nbsp;&nbsp; latency&nbsp;&nbsp; : target=0, window=0, > percentile=100.00%, depth=1 > > Run status group 0 (all jobs): > &nbsp; WRITE: bw=6417KiB/s (6571kB/s), 6417KiB/s-6417KiB/s (6571kB/s- > 6571kB/s), io=188MiB (197MB), run=30003-30003msec > > Disk stats (read/write): > &nbsp;&nbsp;&nbsp; dm-0: ios=31/2, merge=0/0, ticks=342/14, in_queue=356, > util=0.12%, aggrios=33/3, aggrmerge=0/0, aggrticks=390/27, > aggrin_queue=404, aggrutil=0.13% &nbsp; sda: ios=33/3, merge=0/0, > ticks=390/27, in_queue=404, util=0.13% > ==================================== > > Does that make sense? How do you benchmark your Ceph cluster? > Appreciate if you could share your experiences here. > > Thanks! > Tony > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 years, 8 months

2
1
0 0

Nautilus: rbd image stuck unaccessible after VM restart

by islepnev＠gmail.com

Hello, I’m running kvm virtualization with rbd storage, some images on rbd pool become efficiently unusable after VM restart. All I/O to problematic rbd image blocks infinitely. Checked that it is not a permission or locking problem. The bug was silent until we performed a planned restart of few VMs and some of VMs failed to start (kvm process timed out). It could be related to recent upgrades luminous to nautilus or proxmox 5 to 6. Ceph backend is clean, no observable problems, all mons/mgrs/osds up and running. Network is ok. Nothing in logs relevant to the problem. ceph version 14.2.6 (ba51347bdbe28c7c0e2e9172fa2983111137bb60) nautilus (stable) kernel 5.3.13-2-pve #1 SMP PVE 5.3.13-2 (Fri, 24 Jan 2020 09:49:36 +0100) x86_64 GNU/Linux HEALTH_OK No locks: # rbd status rbd-technet/vm-402-disk-0 Watchers: none # rbd status rbd-technet/vm-402-disk-1 Watchers: none Normal image vs problematic: # rbd object-map check rbd-technet/vm-402-disk-0 Object Map Check: 100% complete…done. # rbd object-map check rbd-technet/vm-402-disk-1 ^C disk-0 is good while disk-1 is effectively lost. Command hangs for many minutes with no visible activity, interrupted. rbd export runs without problems, however some data is lost after being imported back (ext4 errors). rbd deep copy worked for me. Copy looks good, no errors. # rbd info rbd-technet/vm-402-disk-1 rbd image 'vm-402-disk-1': size 16 GiB in 4096 objects order 22 (4 MiB objects) snapshot_count: 0 id: c600d06b8b4567 block_name_prefix: rbd_data.c600d06b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Fri Jan 31 17:50:50 2020 access_timestamp: Sat Mar 7 00:30:53 2020 modify_timestamp: Sat Mar 7 00:33:35 2020 journal: c600d06b8b4567 mirroring state: disabled What can be done to debug this problem? Thanks, Ilia.

3 years, 8 months

4
3
1 0

Re: benchmark Ceph

by rainning

Can you post the fio results with the ioengine using libaio? From what you posted, it seems to me that the read test hit cache. And the write performance was not good, the latency was too high (~35.4ms) while the numjobs and iodepth both were 1. Did you monitor system stat on both side (VM/Compute Node and Cluster)? ------------------ Original ------------------ From:  "Tony Liu";<tonyliu0592(a)hotmail.com>gt;; Date:  Sep 15, 2020 To:  "ceph-users"<ceph-users(a)ceph.io>gt;; Subject:  [ceph-users] benchmark Ceph Hi, I have a 3-OSD-node Ceph cluster with 1 480GB SSD and 8 x 2TB 12Gpbs SAS HDD on each node, to provide storage to a OpenStack cluster. Both public and cluster networks are 2x10G. WAL and DB of each OSD is on SSD and they share the same 60GB partition. I run fio with different combinations of operation, block size and io-depth to collect IOPS, bandwidth and latency. I tried fio on compute node with ioengine=rbd, also fio within VM (backed by Ceph) with ioengine=libaio. The result doesn't seem good. Here are couple examples. ==================================== fio --name=test --ioengine=rbd --clientname=admin \     --pool=benchmark --rbdname=test --numjobs=1 \     --runtime=30 --direct=1 --size=2G \     --rw=read --bs=4k --iodepth=1 test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.7 Starting 1 process Jobs: 1 (f=0): [f(1)][100.0%][r=27.6MiB/s,w=0KiB/s][r=7075,w=0 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=56310: Mon Sep 14 19:01:24 2020    read: IOPS=7610, BW=29.7MiB/s (31.2MB/s)(892MiB/30001msec)     slat (nsec): min=1550, max=57662, avg=3312.74, stdev=2981.42     clat (usec): min=77, max=4799, avg=127.39, stdev=39.88      lat (usec): min=78, max=4812, avg=130.70, stdev=40.67     clat percentiles (usec):      |  1.00th=[   82],  5.00th=[   86], 10.00th=[   95], 20.00th=[   98],      | 30.00th=[  100], 40.00th=[  104], 50.00th=[  116], 60.00th=[  129],      | 70.00th=[  141], 80.00th=[  157], 90.00th=[  182], 95.00th=[  198],      | 99.00th=[  233], 99.50th=[  245], 99.90th=[  359], 99.95th=[  515],      | 99.99th=[  709]    bw (  KiB/s): min=27160, max=40696, per=100.00%, avg=30474.29, stdev=2826.23, samples=59    iops        : min= 6790, max=10174, avg=7618.56, stdev=706.56, samples=59   lat (usec)   : 100=28.89%, 250=70.72%, 500=0.34%, 750=0.05%, 1000=0.01%   lat (msec)   : 2=0.01%, 10=0.01%   cpu          : usr=3.55%, sys=3.80%, ctx=228358, majf=0, minf=29   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      issued rwts: total=228333,0,0,0 short=0,0,0,0 dropped=0,0,0,0      latency   : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs):    READ: bw=29.7MiB/s (31.2MB/s), 29.7MiB/s-29.7MiB/s (31.2MB/s-31.2MB/s), io=892MiB (935MB), run=30001-30001msec Disk stats (read/write):     dm-0: ios=290/3, merge=0/0, ticks=2427/19, in_queue=2446, util=0.95%, aggrios=290/4, aggrmerge=0/0, aggrticks=2427/39, aggrin_queue=2332, aggrutil=0.95%   sda: ios=290/4, merge=0/0, ticks=2427/39, in_queue=2332, util=0.95% ==================================== ==================================== fio --name=test --ioengine=rbd --clientname=admin \     --pool=benchmark --rbdname=test --numjobs=1 \     --runtime=30 --direct=1 --size=2G \     --rw=write --bs=4k --iodepth=1 test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.7 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=6352KiB/s][r=0,w=1588 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=56544: Mon Sep 14 19:03:36 2020   write: IOPS=1604, BW=6417KiB/s (6571kB/s)(188MiB/30003msec)     slat (nsec): min=2240, max=45925, avg=6526.95, stdev=3486.19     clat (usec): min=399, max=35411, avg=615.88, stdev=231.41      lat (usec): min=402, max=35421, avg=622.40, stdev=232.08     clat percentiles (usec):      |  1.00th=[  420],  5.00th=[  449], 10.00th=[  469], 20.00th=[  498],      | 30.00th=[  529], 40.00th=[  562], 50.00th=[  611], 60.00th=[  652],      | 70.00th=[  685], 80.00th=[  709], 90.00th=[  766], 95.00th=[  799],      | 99.00th=[  881], 99.50th=[  955], 99.90th=[ 2671], 99.95th=[ 3097],      | 99.99th=[ 3785]    bw (  KiB/s): min= 5944, max= 6792, per=100.00%, avg=6415.95, stdev=178.72, samples=60    iops        : min= 1486, max= 1698, avg=1603.93, stdev=44.67, samples=60   lat (usec)   : 500=20.82%, 750=67.23%, 1000=11.55%   lat (msec)   : 2=0.25%, 4=0.14%, 10=0.01%, 20=0.01%, 50=0.01%   cpu          : usr=1.22%, sys=1.25%, ctx=48143, majf=0, minf=18   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      issued rwts: total=0,48129,0,0 short=0,0,0,0 dropped=0,0,0,0      latency   : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs):   WRITE: bw=6417KiB/s (6571kB/s), 6417KiB/s-6417KiB/s (6571kB/s-6571kB/s), io=188MiB (197MB), run=30003-30003msec Disk stats (read/write):     dm-0: ios=31/2, merge=0/0, ticks=342/14, in_queue=356, util=0.12%, aggrios=33/3, aggrmerge=0/0, aggrticks=390/27, aggrin_queue=404, aggrutil=0.13%   sda: ios=33/3, merge=0/0, ticks=390/27, in_queue=404, util=0.13% ==================================== Does that make sense? How do you benchmark your Ceph cluster? Appreciate if you could share your experiences here. Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 years, 8 months

2
1
0 0

2024

2023

2022

2021

2020

2019

ceph-users September 2020