You have to tell a bit about your cluster setup, like nr of osd's, 3x replication on your testing pool? Eg. this[1] was my test on a cluster with only 1gbit ethernet, 3x repl hdd pool. This[2] with 10gbit and more osd's added [2] [root@c01 ~]# rados bench -p rbd 10 write hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_c01_3576497 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 41 25 99.9948 100 0.198773 0.41148 2 16 101 85 169.984 240 0.203578 0.347027 3 16 172 156 207.979 284 0.0863202 0.296866 4 16 245 229 228.975 292 0.139681 0.268933 5 16 322 306 244.772 308 0.107296 0.257353 6 16 385 369 245.97 252 0.601879 0.250782 7 16 460 444 253.684 300 0.154803 0.247178 8 16 541 525 262.467 324 0.274302 0.241951 9 16 604 588 261.3 252 0.11929 0.238717 10 16 672 656 262.367 272 0.134654 0.241424 Total time run: 10.1504 Total writes made: 673 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 265.212 Stddev Bandwidth: 63.0823 Max bandwidth (MB/sec): 324 Min bandwidth (MB/sec): 100 Average IOPS: 66 Stddev IOPS: 15.7706 Max IOPS: 81 Min IOPS: 25 Average Latency(s): 0.241012 Stddev Latency(s): 0.154282 Max latency(s): 1.05851 Min latency(s): 0.0702826 Cleaning up (deleting benchmark objects) Removed 673 objects Clean up completed and total clean up time :1.26346 [1] [@]# rados bench -p rbd 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_c01_18283 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 27 11 43.9884 44 0.554119 0.624979 2 16 47 31 61.9841 80 1.04112 0.793553 3 16 57 41 54.654 40 1.33104 0.876273 4 16 75 59 58.9869 72 0.840098 0.97091 5 16 97 81 64.7864 88 1.02915 0.922043 6 16 105 89 59.3207 32 1.2471 0.915408 7 16 129 113 64.5582 96 0.616579 0.947882 8 16 145 129 64.4866 64 1.09397 0.921441 9 16 163 147 65.3201 72 0.885566 0.906388 10 16 166 150 59.9881 12 1.22834 0.909591 11 13 167 154 55.9889 16 2.30029 0.942798 Total time run: 11.141939 Total writes made: 167 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 59.9537 Stddev Bandwidth: 28.7889 Max bandwidth (MB/sec): 96 Min bandwidth (MB/sec): 12 Average IOPS: 14 Stddev IOPS: 7 Max IOPS: 24 Min IOPS: 3 Average Latency(s): 1.06157 Stddev Latency(s): 0.615773 Max latency(s): 3.23088 Min latency(s): 0.171585
-----Original Message----- Sent: 10 February 2021 10:14 To: Marc <Marc@f1-outsourcing.eu> Cc: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Re: struggling to achieve high bandwidth on Ceph dev cluster - HELP
thanks for the reply.
Yes, 4MB is the default. I have tried it. For example below (posted) is for 4MB (default) ran for 600 seconds. The seq read and rand read gives me a good bandwidth (not posted here). But with write its still very less. And I am particularly interested in block sizes. And rados bench tool has block size option which I have been using.
Total time run: 601.106 Total writes made: 2966 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 19.7369 Stddev Bandwidth: 14.8408 Max bandwidth (MB/sec): 64 Min bandwidth (MB/sec): 0 Average IOPS: 4 Stddev IOPS: 3.67408 Max IOPS: 16 Min IOPS: 0 Average Latency(s): 3.24064 Stddev Latency(s): 2.75111 Max latency(s): 42.4551 Min latency(s): 0.167701
On Wed, Feb 10, 2021 at 9:46 AM Marc <Marc@f1-outsourcing.eu> wrote:
try 4MB that is the default not?
-----Original Message----- Sent: 10 February 2021 09:30 To: ceph-users <ceph-users@ceph.io>; dev <dev@ceph.io>; ceph-
Subject: [ceph-users] struggling to achieve high bandwidth on Ceph dev cluster - HELP
Hi,
Hello I am using rados bench tool. Currently I am using this tool on the development cluster after running vstart.sh script. It is working fine and I am interested in benchmarking the cluster. However I am struggling to achieve a good bandwidth i.e. bandwidth (MB/sec). My target
qa@ceph.io throughput
is at least 50 MB/sec and more. But mostly I am achieving is around 15- 20 MB/sec. So, very poor.
I am quite sure I am missing something. Either I have to change my cluster through vstart.sh script or I am not fully utilizing the rados bench tool. Or may be both. i.e. not the right cluster and also not using the rados bench tool correctly.
Some of the shell examples I have been using to build the cluster are bellow: MDS=0 RGW=1 ../src/vstart.sh -d -l -n --bluestore MDS=0 RGW=1 MON=1 OSD=4../src/vstart.sh -d -l -n --bluestore
While using rados bench tool I have been trying with different block sizes 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K. And I have also been changing the -t parameter in the shell to increase concurrent IOs.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io