RBD Performance issue
Hi Team, I have set up a 2 Ceph cluster 3-node each cluster with a two-way RBD mirror. In this setup, Ceph 1 is configured two-way mirror to Ceph 2, and vice versa. The RBD pools are integrated with CloudStack. The Ceph cluster uses NVMe drives, but I am experiencing very low IOPS performance. I have provided the relevant details below. Could you please guide me on how to optimize the setup to achieve higher IOPS? fio --ioengine=libaio --direct=1 --randrepeat=1 --refill_buffers --end_fsync=1 --rwmixread=70 --filename=/root/ceph-rbd --name=write --size=1024m --bs=4k --rw=readwrite --iodepth=32 --numjobs=16 --group_reporting read: IOPS=92.5k, BW=361MiB/s (379MB/s)(11.2GiB/31718msec) write: IOPS=39.7k, BW=155MiB/s (163MB/s)(4922MiB/31718msec); 0 zone resets Hardware Specifications: - CPU: Intel(R) Xeon(R) Gold 5416S - RAM: 125 GB - Storage: 8 x 7 TB NVMe disks (Model: UP2A67T6SD004LX) [Drive specifications](https://www.techpowerup.com/ssd-specs/union-memory-uh711a-7-5-tb.d1802) - Network: 4 x 25 Gbps interfaces configured with LACP bonding Each server in the setup is equipped with the above configuration.
You mention RBD, but you give FIO a filename. Are you writing to a file on filesystem on an RBD volume? Are you testing from a VM? From one of the cluster nodes? Via a KRBD mount? Do you get better results with the volume unattached and using the librbd ioengine? What does the rbd mirror status for this volume look like? Is it far behind? Which rbd-mirror mode are you using? Send `ceph osd dump | grep pool` and `ceph -s` for both clusters.
On Feb 17, 2025, at 12:38 AM, vignesh varma <vignesh.varma.g@stackbill.com> wrote:
Hi Team,
I have set up a 2 Ceph cluster 3-node each cluster with a two-way RBD mirror. In this setup, Ceph 1 is configured two-way mirror to Ceph 2, and vice versa. The RBD pools are integrated with CloudStack.
The Ceph cluster uses NVMe drives, but I am experiencing very low IOPS performance. I have provided the relevant details below. Could you please guide me on how to optimize the setup to achieve higher IOPS?
fio --ioengine=libaio --direct=1 --randrepeat=1 --refill_buffers --end_fsync=1 --rwmixread=70 --filename=/root/ceph-rbd --name=write --size=1024m --bs=4k --rw=readwrite --iodepth=32 --numjobs=16 --group_reporting
read: IOPS=92.5k, BW=361MiB/s (379MB/s)(11.2GiB/31718msec) write: IOPS=39.7k, BW=155MiB/s (163MB/s)(4922MiB/31718msec); 0 zone resets
Hardware Specifications:
- CPU: Intel(R) Xeon(R) Gold 5416S - RAM: 125 GB - Storage: 8 x 7 TB NVMe disks (Model: UP2A67T6SD004LX) [Drive specifications](https://www.techpowerup.com/ssd-specs/union-memory-uh711a-7-5-tb.d1802) - Network: 4 x 25 Gbps interfaces configured with LACP bonding
Each server in the setup is equipped with the above configuration. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi darren & Anthony,
How many PG’s have you got configured for the Ceph pool that you are testing against? I have crealed the cloudstack pools with the PG of 64.
ceph osd pool get cloudstack-BRK pg_autoscale_mode pg_autoscale_mode: on
Have you tried the same benchmark without the replication setup? NO, I have connected the 3repliction pool to cloudstack
RBD bech With replican 1 rados bench -p testbench 10 write --run-name client1 hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_STR-01-BRK_342365 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 860 844 3375.75 3376 0.00919862 0.0187299 2 16 1775 1759 3517.55 3660 0.0128823 0.0180904 3 16 2773 2757 3675.46 3992 0.0193005 0.0173626 4 16 3759 3743 3742.45 3944 0.00819503 0.0170552 5 16 4743 4727 3781.04 3936 0.0202672 0.0169013 6 16 5766 5750 3832.77 4092 0.0124358 0.0166735 7 16 6783 6767 3866.28 4068 0.0235515 0.0165316 8 16 7761 7745 3871.91 3912 0.0162872 0.0165084 9 16 8791 8775 3899.39 4120 0.0159618 0.0163925 10 16 9810 9794 3916.99 4076 0.0267244 0.0163123 Total time run: 10.0108 Total writes made: 9810 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 3919.78 Stddev Bandwidth: 232.231 Max bandwidth (MB/sec): 4120 Min bandwidth (MB/sec): 3376 Average IOPS: 979 Stddev IOPS: 58.0578 Max IOPS: 1030 Min IOPS: 844 Average Latency(s): 0.0163137 Stddev Latency(s): 0.00699291 Max latency(s): 0.0683829 Min latency(s): 0.00599489 Cleaning up (deleting benchmark objects) Removed 9810 objects Clean up completed and total clean up time :0.47255 rbd bench image01 --pool=testbench --io-size=256M --io-threads=8 --io-type=write bench type write io_size 268435456 io_threads 8 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC elapsed: 5 ops: 4 ops/sec: 0.71125 bytes/sec: 182 MiB/s With replican 3 rados bench -p rbd_rnd 10 write --run-name client1 hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_STR-01-BRK_342454 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 421 405 1619.88 1620 0.035311 0.0386264 2 16 832 816 1631.76 1644 0.0235547 0.0386749 3 16 1246 1230 1639.73 1656 0.0367235 0.0388173 4 16 1660 1644 1643.72 1656 0.0444481 0.0387138 5 16 2090 2074 1658.92 1720 0.0296465 0.0384273 6 16 2515 2499 1665.72 1700 0.0474488 0.0383093 7 16 2928 2912 1663.72 1652 0.0404621 0.0383227 8 16 3356 3340 1669.72 1712 0.0346635 0.0382409 9 16 3781 3765 1673.04 1700 0.0356325 0.0381586 10 14 4196 4182 1672.51 1668 0.0455228 0.0381928 Total time run: 10.0196 Total writes made: 4196 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 1675.11 Stddev Bandwidth: 33.1354 Max bandwidth (MB/sec): 1720 Min bandwidth (MB/sec): 1620 Average IOPS: 418 Stddev IOPS: 8.28385 Max IOPS: 430 Min IOPS: 405 Average Latency(s): 0.0381648 Stddev Latency(s): 0.011959 Max latency(s): 0.100015 Min latency(s): 0.0172346 Cleaning up (deleting benchmark objects) Removed 4196 objects Clean up completed and total clean up time :0.357656 rbd bench image01 --pool=rbd_rnd --io-size=256M --io-threads=8 --io-type=write bench type write io_size 268435456 io_threads 8 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC elapsed: 5 ops: 4 ops/sec: 0.692054 bytes/sec: 177 MiB/s
Which replication are you using snapshot or journal based?
snapshot based replication
Can you run a benchmark with 100% read and 100% write to see what the differences are between them?
fio --ioengine=libaio --direct=1 --randrepeat=1 --refill_buffers --end_fsync=1 --filename=/root/ceph-rbd --name=write --size=1024m --bs=4k --rw=write --iodepth=32 --numjobs=16 write: IOPS=7840, BW=30.6MiB/s (32.1MB/s)(1024MiB/33434msec); 0 zone resets fio --ioengine=libaio --direct=1 --randrepeat=1 --refill_buffers --end_fsync=1 --filename=/root/ceph-rbd --name=read --size=1024m --bs=4k --rw=read --iodepth=32 --numjobs=16 read: IOPS=13.8k, BW=53.9MiB/s (56.5MB/s)(1024MiB/18990msec)
How have you configured data protection? Erasure coded or Replicas?
Is the machine you are running the benchmark on a cloud stack virtual machine or a bare
Replicas metal physical machine? Inside the virtual machine deployed via cloud stack
What is the network connectivity of the machine you are benchmarking on?
25*4 GIB NIC with LACP bond for public and cluster network
Send `ceph osd dump | grep pool` and `ceph -s` for both clusters.
Cluster 1 ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 35 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 23.08 pool 2 'cloudstack-GUL' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 79031 lfor 0/0/85 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.63 pool 3 'cloudstack-BRK' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 78978 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.25 pool 4 '.nfs' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 105 lfor 0/0/103 flags hashpspool stripe_width 0 application nfs read_balance_score 3.00 pool 5 'cephfs.cephfs.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 144 lfor 0/0/114 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 4.48 pool 6 'cephfs.cephfs.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 144 lfor 0/0/142 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 1.27 pool 7 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 3262 lfor 0/0/3260 flags hashpspool stripe_width 0 application rgw read_balance_score 3.00 pool 8 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 17306 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.88 ceph -s cluster: id: 1614bf56-70fe-11ef-890c-8d560ac5fab0 health: HEALTH_OK services: mon: 3 daemons, quorum STR-01-GUL,STR-02-GUL,STR-03-GUL (age 2M) mgr: STR-01-GUL.lovayi(active, since 2M), standbys: STR-02-GUL.jxryby mds: 1/1 daemons up, 1 standby osd: 24 osds: 24 up (since 5M), 24 in (since 5M) rbd-mirror: 1 daemon active (1 hosts) data: volumes: 1/1 healthy pools: 8 pools, 1297 pgs objects: 1.07M objects, 3.9 TiB usage: 12 TiB used, 156 TiB / 168 TiB avail pgs: 1297 active+clean io: client: 2.9 MiB/s rd, 7.8 MiB/s wr, 376 op/s rd, 608 op/s wr Cluster 2 ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 35 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 23.08 pool 2 'cloudstack-GUL' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 85862 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.63 pool 3 'cloudstack-BRK' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 85791 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.88 pool 4 '.nfs' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 97 lfor 0/0/95 flags hashpspool stripe_width 0 application nfs read_balance_score 3.00 pool 5 'cephfs.cephfs.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 143 lfor 0/0/113 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.99 pool 6 'cephfs.cephfs.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 143 lfor 0/0/141 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 1.27 pool 9 'rbd_rnd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 85857 lfor 0/0/85639 flags hashpspool,selfmanaged_snaps max_bytes 536870912000 stripe_width 0 application rbd read_balance_score 3.00 pool 11 'testbench' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 85838 lfor 0/0/85824 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 3.71 ceph -s cluster: id: d12f0e2a-7100-11ef-ab63-11f6a8d75f5b health: HEALTH_WARN 1 pool(s) have no replicas configured services: mon: 3 daemons, quorum STR-01-BRK,STR-03-BRK,STR-02-BRK (age 2M) mgr: STR-01-BRK.eppard(active, since 5M), standbys: STR-03-BRK.varwpu mds: 1/1 daemons up, 1 standby osd: 24 osds: 24 up (since 5M), 24 in (since 5M) rbd-mirror: 1 daemon active (1 hosts) data: volumes: 1/1 healthy pools: 8 pools, 1265 pgs objects: 329.79k objects, 882 GiB usage: 2.6 TiB used, 165 TiB / 168 TiB avail pgs: 1265 active+clean io: client: 3.7 MiB/s rd, 1.8 MiB/s wr, 234 op/s rd, 181 op/s wr
Hi Darren & Anthony
How many PG’s have you got configured for the Ceph pool that you are testing against? I have crealed the cloudstack pools with the PG of 64.
ceph osd pool get cloudstack-BRK pg_autoscale_mode pg_autoscale_mode: on ceph osd pool ls .mgr cloudstack-GUL cloudstack-BRK .nfs cephfs.cephfs.meta cephfs.cephfs.data
Have you tried the same benchmark without the replication setup? NO, I have connected the 3repliction pool to cloudstack
RBD bech With replican 1 rados bench -p testbench 10 write --run-name client1 hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_STR-01-BRK_342365 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 860 844 3375.75 3376 0.00919862 0.0187299 2 16 1775 1759 3517.55 3660 0.0128823 0.0180904 3 16 2773 2757 3675.46 3992 0.0193005 0.0173626 4 16 3759 3743 3742.45 3944 0.00819503 0.0170552 5 16 4743 4727 3781.04 3936 0.0202672 0.0169013 6 16 5766 5750 3832.77 4092 0.0124358 0.0166735 7 16 6783 6767 3866.28 4068 0.0235515 0.0165316 8 16 7761 7745 3871.91 3912 0.0162872 0.0165084 9 16 8791 8775 3899.39 4120 0.0159618 0.0163925 10 16 9810 9794 3916.99 4076 0.0267244 0.0163123 Total time run: 10.0108 Total writes made: 9810 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 3919.78 Stddev Bandwidth: 232.231 Max bandwidth (MB/sec): 4120 Min bandwidth (MB/sec): 3376 Average IOPS: 979 Stddev IOPS: 58.0578 Max IOPS: 1030 Min IOPS: 844 Average Latency(s): 0.0163137 Stddev Latency(s): 0.00699291 Max latency(s): 0.0683829 Min latency(s): 0.00599489 Cleaning up (deleting benchmark objects) Removed 9810 objects Clean up completed and total clean up time :0.47255 rbd bench image01 --pool=testbench --io-size=256M --io-threads=8 --io-type=write bench type write io_size 268435456 io_threads 8 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC elapsed: 5 ops: 4 ops/sec: 0.71125 bytes/sec: 182 MiB/s With replican 3 rados bench -p rbd_rnd 10 write --run-name client1 hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_STR-01-BRK_342454 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 421 405 1619.88 1620 0.035311 0.0386264 2 16 832 816 1631.76 1644 0.0235547 0.0386749 3 16 1246 1230 1639.73 1656 0.0367235 0.0388173 4 16 1660 1644 1643.72 1656 0.0444481 0.0387138 5 16 2090 2074 1658.92 1720 0.0296465 0.0384273 6 16 2515 2499 1665.72 1700 0.0474488 0.0383093 7 16 2928 2912 1663.72 1652 0.0404621 0.0383227 8 16 3356 3340 1669.72 1712 0.0346635 0.0382409 9 16 3781 3765 1673.04 1700 0.0356325 0.0381586 10 14 4196 4182 1672.51 1668 0.0455228 0.0381928 Total time run: 10.0196 Total writes made: 4196 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 1675.11 Stddev Bandwidth: 33.1354 Max bandwidth (MB/sec): 1720 Min bandwidth (MB/sec): 1620 Average IOPS: 418 Stddev IOPS: 8.28385 Max IOPS: 430 Min IOPS: 405 Average Latency(s): 0.0381648 Stddev Latency(s): 0.011959 Max latency(s): 0.100015 Min latency(s): 0.0172346 Cleaning up (deleting benchmark objects) Removed 4196 objects Clean up completed and total clean up time :0.357656 rbd bench image01 --pool=rbd_rnd --io-size=256M --io-threads=8 --io-type=write bench type write io_size 268435456 io_threads 8 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC elapsed: 5 ops: 4 ops/sec: 0.692054 bytes/sec: 177 MiB/s
Which replication are you using snapshot or journal based?
snapshot based replication
Can you run a benchmark with 100% read and 100% write to see what the differences are between them?
fio --ioengine=libaio --direct=1 --randrepeat=1 --refill_buffers --end_fsync=1 --filename=/root/ceph-rbd --name=write --size=1024m --bs=4k --rw=write --iodepth=32 --numjobs=16 write: IOPS=7840, BW=30.6MiB/s (32.1MB/s)(1024MiB/33434msec); 0 zone resets fio --ioengine=libaio --direct=1 --randrepeat=1 --refill_buffers --end_fsync=1 --filename=/root/ceph-rbd --name=read --size=1024m --bs=4k --rw=read --iodepth=32 --numjobs=16 read: IOPS=13.8k, BW=53.9MiB/s (56.5MB/s)(1024MiB/18990msec)
How have you configured data protection? Erasure coded or Replicas?
Is the machine you are running the benchmark on a cloud stack virtual machine or a bare
Replicas metal physical machine? Inside the virtual machine deployed via cloud stack
What is the network connectivity of the machine you are benchmarking on?
25*4 GIB NIC with LACP bond for public and cluster network
Send `ceph osd dump | grep pool` and `ceph -s` for both clusters.
Cluster 1 ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 35 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 23.08 pool 2 'cloudstack-GUL' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 79031 lfor 0/0/85 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.63 pool 3 'cloudstack-BRK' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 78978 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.25 pool 4 '.nfs' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 105 lfor 0/0/103 flags hashpspool stripe_width 0 application nfs read_balance_score 3.00 pool 5 'cephfs.cephfs.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 144 lfor 0/0/114 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 4.48 pool 6 'cephfs.cephfs.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 144 lfor 0/0/142 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 1.27 pool 7 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 3262 lfor 0/0/3260 flags hashpspool stripe_width 0 application rgw read_balance_score 3.00 pool 8 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 17306 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.88 ceph -s cluster: id: 1614bf56-70fe-11ef-890c-8d560ac5fab0 health: HEALTH_OK services: mon: 3 daemons, quorum STR-01-GUL,STR-02-GUL,STR-03-GUL (age 2M) mgr: STR-01-GUL.lovayi(active, since 2M), standbys: STR-02-GUL.jxryby mds: 1/1 daemons up, 1 standby osd: 24 osds: 24 up (since 5M), 24 in (since 5M) rbd-mirror: 1 daemon active (1 hosts) data: volumes: 1/1 healthy pools: 8 pools, 1297 pgs objects: 1.07M objects, 3.9 TiB usage: 12 TiB used, 156 TiB / 168 TiB avail pgs: 1297 active+clean io: client: 2.9 MiB/s rd, 7.8 MiB/s wr, 376 op/s rd, 608 op/s wr Cluster 2 ceph osd dump | grep pool pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 35 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 23.08 pool 2 'cloudstack-GUL' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 85862 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.63 pool 3 'cloudstack-BRK' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 85791 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.88 pool 4 '.nfs' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 97 lfor 0/0/95 flags hashpspool stripe_width 0 application nfs read_balance_score 3.00 pool 5 'cephfs.cephfs.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 143 lfor 0/0/113 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.99 pool 6 'cephfs.cephfs.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 143 lfor 0/0/141 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 1.27 pool 9 'rbd_rnd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 85857 lfor 0/0/85639 flags hashpspool,selfmanaged_snaps max_bytes 536870912000 stripe_width 0 application rbd read_balance_score 3.00 pool 11 'testbench' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 85838 lfor 0/0/85824 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 3.71 ceph -s cluster: id: d12f0e2a-7100-11ef-ab63-11f6a8d75f5b health: HEALTH_WARN 1 pool(s) have no replicas configured services: mon: 3 daemons, quorum STR-01-BRK,STR-03-BRK,STR-02-BRK (age 2M) mgr: STR-01-BRK.eppard(active, since 5M), standbys: STR-03-BRK.varwpu mds: 1/1 daemons up, 1 standby osd: 24 osds: 24 up (since 5M), 24 in (since 5M) rbd-mirror: 1 daemon active (1 hosts) data: volumes: 1/1 healthy pools: 8 pools, 1265 pgs objects: 329.79k objects, 882 GiB usage: 2.6 TiB used, 165 TiB / 168 TiB avail pgs: 1265 active+clean io: client: 3.7 MiB/s rd, 1.8 MiB/s wr, 234 op/s rd, 181 op/s wr
Hi Vignesh, So a few questions :- How many PG’s have you got configured for the Ceph pool that you are testing against? If this number is not large enough then you may only be using a subset of the devices available to use. Have you tried the same benchmark without the replication setup? Trying to see if the extra writes are slowing things down. Which replication are you using snapshot or journal based? This can generate a lot of extra disk traffic hence the reason for trying the benchmark without replication Can you run a benchmark with 100% read and 100% write to see what the differences are between them? Trying to see if reads or writes could be slowing the other one down. How have you configured data protection? Erasure coded or Replicas? Is the machine you are running the benchmark on a cloud stack virtual machine or a bare metal physical machine? What is the network connectivity of the machine you are benchmarking on?
On 17 Feb 2025, at 05:38, vignesh varma <vignesh.varma.g@stackbill.com> wrote:
Hi Team,
I have set up a 2 Ceph cluster 3-node each cluster with a two-way RBD mirror. In this setup, Ceph 1 is configured two-way mirror to Ceph 2, and vice versa. The RBD pools are integrated with CloudStack.
The Ceph cluster uses NVMe drives, but I am experiencing very low IOPS performance. I have provided the relevant details below. Could you please guide me on how to optimize the setup to achieve higher IOPS?
fio --ioengine=libaio --direct=1 --randrepeat=1 --refill_buffers --end_fsync=1 --rwmixread=70 --filename=/root/ceph-rbd --name=write --size=1024m --bs=4k --rw=readwrite --iodepth=32 --numjobs=16 --group_reporting
read: IOPS=92.5k, BW=361MiB/s (379MB/s)(11.2GiB/31718msec) write: IOPS=39.7k, BW=155MiB/s (163MB/s)(4922MiB/31718msec); 0 zone resets
Hardware Specifications:
- CPU: Intel(R) Xeon(R) Gold 5416S - RAM: 125 GB - Storage: 8 x 7 TB NVMe disks (Model: UP2A67T6SD004LX) [Drive specifications](https://www.techpowerup.com/ssd-specs/union-memory-uh711a-7-5-tb.d1802) - Network: 4 x 25 Gbps interfaces configured with LACP bonding
Each server in the setup is equipped with the above configuration. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Anthony D'Atri
-
darren＠soothill.com
-
vignesh varma