Hi Darren & Anthony
How many PG’s have you got configured for the Ceph pool that you are testing against? I have crealed the cloudstack pools with the PG of 64.
ceph osd pool get cloudstack-BRK pg_autoscale_mode pg_autoscale_mode: on ceph osd pool ls .mgr cloudstack-GUL cloudstack-BRK .nfs cephfs.cephfs.meta cephfs.cephfs.data
Have you tried the same benchmark without the replication setup? NO, I have connected the 3repliction pool to cloudstack
RBD bech With replican 1 rados bench -p testbench 10 write --run-name client1 hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_STR-01-BRK_342365 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 860 844 3375.75 3376 0.00919862 0.0187299 2 16 1775 1759 3517.55 3660 0.0128823 0.0180904 3 16 2773 2757 3675.46 3992 0.0193005 0.0173626 4 16 3759 3743 3742.45 3944 0.00819503 0.0170552 5 16 4743 4727 3781.04 3936 0.0202672 0.0169013 6 16 5766 5750 3832.77 4092 0.0124358 0.0166735 7 16 6783 6767 3866.28 4068 0.0235515 0.0165316 8 16 7761 7745 3871.91 3912 0.0162872 0.0165084 9 16 8791 8775 3899.39 4120 0.0159618 0.0163925 10 16 9810 9794 3916.99 4076 0.0267244 0.0163123 Total time run: 10.0108 Total writes made: 9810 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 3919.78 Stddev Bandwidth: 232.231 Max bandwidth (MB/sec): 4120 Min bandwidth (MB/sec): 3376 Average IOPS: 979 Stddev IOPS: 58.0578 Max IOPS: 1030 Min IOPS: 844 Average Latency(s): 0.0163137 Stddev Latency(s): 0.00699291 Max latency(s): 0.0683829 Min latency(s): 0.00599489 Cleaning up (deleting benchmark objects) Removed 9810 objects Clean up completed and total clean up time :0.47255 rbd bench image01 --pool=testbench --io-size=256M --io-threads=8 --io-type=write bench type write io_size 268435456 io_threads 8 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC elapsed: 5 ops: 4 ops/sec: 0.71125 bytes/sec: 182 MiB/s With replican 3 rados bench -p rbd_rnd 10 write --run-name client1 hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_STR-01-BRK_342454 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 421 405 1619.88 1620 0.035311 0.0386264 2 16 832 816 1631.76 1644 0.0235547 0.0386749 3 16 1246 1230 1639.73 1656 0.0367235 0.0388173 4 16 1660 1644 1643.72 1656 0.0444481 0.0387138 5 16 2090 2074 1658.92 1720 0.0296465 0.0384273 6 16 2515 2499 1665.72 1700 0.0474488 0.0383093 7 16 2928 2912 1663.72 1652 0.0404621 0.0383227 8 16 3356 3340 1669.72 1712 0.0346635 0.0382409 9 16 3781 3765 1673.04 1700 0.0356325 0.0381586 10 14 4196 4182 1672.51 1668 0.0455228 0.0381928 Total time run: 10.0196 Total writes made: 4196 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 1675.11 Stddev Bandwidth: 33.1354 Max bandwidth (MB/sec): 1720 Min bandwidth (MB/sec): 1620 Average IOPS: 418 Stddev IOPS: 8.28385 Max IOPS: 430 Min IOPS: 405 Average Latency(s): 0.0381648 Stddev Latency(s): 0.011959 Max latency(s): 0.100015 Min latency(s): 0.0172346 Cleaning up (deleting benchmark objects) Removed 4196 objects Clean up completed and total clean up time :0.357656 rbd bench image01 --pool=rbd_rnd --io-size=256M --io-threads=8 --io-type=write bench type write io_size 268435456 io_threads 8 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC elapsed: 5 ops: 4 ops/sec: 0.692054 bytes/sec: 177 MiB/s
Which replication are you using snapshot or journal based?
snapshot based replication
Can you run a benchmark with 100% read and 100% write to see what the differences are between them?
fio --ioengine=libaio --direct=1 --randrepeat=1 --refill_buffers --end_fsync=1 --filename=/root/ceph-rbd --name=write --size=1024m --bs=4k --rw=write --iodepth=32 --numjobs=16 write: IOPS=7840, BW=30.6MiB/s (32.1MB/s)(1024MiB/33434msec); 0 zone resets fio --ioengine=libaio --direct=1 --randrepeat=1 --refill_buffers --end_fsync=1 --filename=/root/ceph-rbd --name=read --size=1024m --bs=4k --rw=read --iodepth=32 --numjobs=16 read: IOPS=13.8k, BW=53.9MiB/s (56.5MB/s)(1024MiB/18990msec)
How have you configured data protection? Erasure coded or Replicas?
Is the machine you are running the benchmark on a cloud stack virtual machine or a bare
Replicas metal physical machine? Inside the virtual machine deployed via cloud stack
What is the network connectivity of the machine you are benchmarking on?
25*4 GIB NIC with LACP bond for public and cluster network
Send `ceph osd dump | grep pool` and `ceph -s` for both clusters.
Cluster 1 ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 35 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 23.08 pool 2 'cloudstack-GUL' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 79031 lfor 0/0/85 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.63 pool 3 'cloudstack-BRK' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 78978 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.25 pool 4 '.nfs' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 105 lfor 0/0/103 flags hashpspool stripe_width 0 application nfs read_balance_score 3.00 pool 5 'cephfs.cephfs.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 144 lfor 0/0/114 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 4.48 pool 6 'cephfs.cephfs.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 144 lfor 0/0/142 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 1.27 pool 7 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 3262 lfor 0/0/3260 flags hashpspool stripe_width 0 application rgw read_balance_score 3.00 pool 8 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 17306 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.88 ceph -s cluster: id: 1614bf56-70fe-11ef-890c-8d560ac5fab0 health: HEALTH_OK services: mon: 3 daemons, quorum STR-01-GUL,STR-02-GUL,STR-03-GUL (age 2M) mgr: STR-01-GUL.lovayi(active, since 2M), standbys: STR-02-GUL.jxryby mds: 1/1 daemons up, 1 standby osd: 24 osds: 24 up (since 5M), 24 in (since 5M) rbd-mirror: 1 daemon active (1 hosts) data: volumes: 1/1 healthy pools: 8 pools, 1297 pgs objects: 1.07M objects, 3.9 TiB usage: 12 TiB used, 156 TiB / 168 TiB avail pgs: 1297 active+clean io: client: 2.9 MiB/s rd, 7.8 MiB/s wr, 376 op/s rd, 608 op/s wr Cluster 2 ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 35 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 23.08 pool 2 'cloudstack-GUL' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 85862 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.63 pool 3 'cloudstack-BRK' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 85791 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.88 pool 4 '.nfs' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 97 lfor 0/0/95 flags hashpspool stripe_width 0 application nfs read_balance_score 3.00 pool 5 'cephfs.cephfs.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 143 lfor 0/0/113 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.99 pool 6 'cephfs.cephfs.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 143 lfor 0/0/141 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 1.27 pool 9 'rbd_rnd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 85857 lfor 0/0/85639 flags hashpspool,selfmanaged_snaps max_bytes 536870912000 stripe_width 0 application rbd read_balance_score 3.00 pool 11 'testbench' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 85838 lfor 0/0/85824 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 3.71 ceph -s cluster: id: d12f0e2a-7100-11ef-ab63-11f6a8d75f5b health: HEALTH_WARN 1 pool(s) have no replicas configured services: mon: 3 daemons, quorum STR-01-BRK,STR-03-BRK,STR-02-BRK (age 2M) mgr: STR-01-BRK.eppard(active, since 5M), standbys: STR-03-BRK.varwpu mds: 1/1 daemons up, 1 standby osd: 24 osds: 24 up (since 5M), 24 in (since 5M) rbd-mirror: 1 daemon active (1 hosts) data: volumes: 1/1 healthy pools: 8 pools, 1265 pgs objects: 329.79k objects, 882 GiB usage: 2.6 TiB used, 165 TiB / 168 TiB avail pgs: 1265 active+clean io: client: 3.7 MiB/s rd, 1.8 MiB/s wr, 234 op/s rd, 181 op/s wr