21 Mar
2023
21 Mar
'23
1:10 p.m.
Hi,
we are currently testing out ways to increase Ceph performance because
what we experience so far is very close to unusable.
For the test cluster we utilizing 4 nodes with the following hardware data:
Dual 200GBe Mellanox Ethernet
2x EPYC Rome 7302
16x 32GB 3200MHz ECC
9x 15.36TB Micron 9300 Pro
For production this will be extended to all 8 nodes if it shows
promising results.
- Ceph was installed with cephadm.
- MDS and ODS are located on the same nodes.
- Mostly using stock config
- Network performance tested with iperf3 seems fine, 26Gbits/s with -P4
on single port (details below).
Close to 200Gbits with 10 parallel instances and servers.
When testing a mounted CephFS on the working nodes in various
configurations I only got <50MB/s for fuse mount and <270MB/s for kernel
mounts. (dd command and output attached below)
In addition ceph dashboard and our graphana monitoring reports packet
loss on all relevant interfaces during load. Which does not occur during
the normal iperf load tests or rsync/scp file transfer.
Rados Bench shows performance around 2000MB/s which is not max
performance of the SSDs but fine for us (details below).
Why is the filesystem so slow compared to the individual components?
Cheers
Dominik
Test details:
------------------------------------------------------------------------------------------------------
Some tests done on working nodes:
Ceph mounted with ceph-fuse
root@ml2ran10:/mnt/cephfs/backup# dd if=/dev/zero of=testfile bs=1M
count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4,3 GB, 4,0 GiB) copied, 88,2933 s, 48,6 MB/s
Ceph mounted with kernel driver:
root@ml2ran06:/mnt/cephfs/backup# dd if=/dev/zero of=testfile bs=1M
count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.0989 s, 267 MB/s
Storage Node
With fuse
root@ml2rsn05:/mnt/ml2r_storage/backup# dd if=/dev/zero of=testfile
bs=1M count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 53.9977 s, 79.5 MB/s
Kernel mount:
dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.6726 s, 243 MB/s
_______________________________________________________
Iperf3
iperf3 --zerocopy -n 10240M -P4 -c ml2ran08s0 -p 4701 -i 15 -b 200000000000
Connecting to host ml2ran08s0, port 4701
[ 5] local 129.217.31.180 port 43958 connected to 129.217.31.218 port 4701
[ 7] local 129.217.31.180 port 43960 connected to 129.217.31.218 port 4701
[ 9] local 129.217.31.180 port 43962 connected to 129.217.31.218 port 4701
[ 11] local 129.217.31.180 port 43964 connected to 129.217.31.218 port 4701
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 632 KBytes
[ 7] 0.00-3.21 sec 2.50 GBytes 6.70 Gbits/sec 0 522 KBytes
[ 9] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 612 KBytes
[ 11] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 430 KBytes
[SUM] 0.00-3.21 sec 10.0 GBytes 26.8 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender
[ 5] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec
receiver
[ 7] 0.00-3.21 sec 2.50 GBytes 6.70 Gbits/sec 0 sender
[ 7] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec
receiver
[ 9] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender
[ 9] 0.00-3.21 sec 2.49 GBytes 6.67 Gbits/sec
receiver
[ 11] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender
[ 11] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec
receiver
[SUM] 0.00-3.21 sec 10.0 GBytes 26.8 Gbits/sec 0 sender
[SUM] 0.00-3.21 sec 9.98 GBytes 26.7 Gbits/sec
receiver
_________________________________________________________________
Rados Bench on storage node:
# rados bench -p testbench 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size
4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_ml2rsn05_2829244
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 0 0 0 0 0 - 0
1 16 747 731 2923.84 2924 0.0178757 0.0216262
2 16 1506 1490 2979.71 3036 0.0308664 0.0213685
3 16 2267 2251 3000.99 3044 0.0259053 0.0212556
4 16 3058 3042 3041.62 3164 0.0227621 0.0209792
5 16 3850 3834 3066.8 3168 0.0130519 0.0208148
6 16 4625 4609 3072.26 3100 0.151371 0.0207904
7 16 5381 5365 3065.28 3024 0.0300368 0.0208345
8 16 6172 6156 3077.57 3164 0.0197728 0.0207714
9 16 6971 6955 3090.67 3196 0.0142751 0.0206786
10 14 7772 7758 3102.76 3212 0.0181034 0.020605
Total time run: 10.0179
Total writes made: 7772
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 3103.23
Stddev Bandwidth: 93.3676
Max bandwidth (MB/sec): 3212
Min bandwidth (MB/sec): 2924
Average IOPS: 775
Stddev IOPS: 23.3419
Max IOPS: 803
Min IOPS: 731
Average Latency(s): 0.020598
Stddev Latency(s): 0.00731743
Max latency(s): 0.151371
Min latency(s): 0.00966991
# rados bench -p testbench 10 seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 0 0 0 0 0 - 0
1 15 657 642 2567.32 2568 0.011104 0.022631
2 15 1244 1229 2456.9 2348 0.0115248 0.019485
3 16 1499 1483 1976.64 1016 0.00722983
0.0177887
4 15 1922 1907 1906.16 1696 0.0142242 0.0327382
5 15 2593 2578 2061.6 2684 0.011758 0.0301774
6 16 3142 3126 2083.23 2192 0.00915926
0.027478
7 16 3276 3260 1862.23 536 0.00824714
0.0267449
8 16 3606 3590 1794.43 1320 0.0118938 0.0350541
9 16 4293 4277 1900.32 2748 0.0301886 0.0330604
10 14 5003 4989 1995.04 2848 0.0389717 0.0314977
Total time run: 10.0227
Total reads made: 5003
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 1996.67
Average IOPS: 499
Stddev IOPS: 202.3
Max IOPS: 712
Min IOPS: 134
Average Latency(s): 0.0314843
Max latency(s): 3.04463
Min latency(s): 0.00551523
# rados bench -p testbench 10 rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 15 15 0 0 0 - 0
1 15 680 665 2657.61 2660 0.00919807
0.0224833
2 15 1273 1258 2514.26 2372 0.00839656
0.0247125
3 16 1863 1847 2461.4 2356 0.00994467
0.0236565
4 16 2064 2048 2047.14 804 0.00809139
0.0223506
5 16 2064 2048 1637.79 0 - 0.0223506
6 16 2477 2461 1640.12 826 0.0286315 0.0383254
7 16 3102 3086 1762.89 2500 0.0267464 0.0349189
8 16 3513 3497 1748 1644 0.00890952
0.032269
9 16 3617 3601 1600 416 0.00626917
0.0316019
10 15 4014 3999 1599.18 1592 0.0461076 0.0393606
Total time run: 10.0481
Total reads made: 4014
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 1597.91
Average IOPS: 399
Stddev IOPS: 239.089
Max IOPS: 665
Min IOPS: 0
Average Latency(s): 0.0394035
Max latency(s): 3.00962
Min latency(s): 0.00449537
22 Mar
22 Mar
7:37 p.m.
Hi Dominik,
if you need performance, the default configuration is not designed for that.
When tuning performance, please pay attention to the sources, if
benchmark or similar is described there, you must pay attention to
whether it is suitable for productive operation.
Regards, Joachim
___________________________________
Clyso GmbH - Ceph Foundation Member
Am 21.03.23 um 10:40 schrieb Dominik Baack:
> Hi,
>
> we are currently testing out ways to increase Ceph performance because
> what we experience so far is very close to unusable.
>
> For the test cluster we utilizing 4 nodes with the following hardware
> data:
>
> Dual 200GBe Mellanox Ethernet
> 2x EPYC Rome 7302
> 16x 32GB 3200MHz ECC
> 9x 15.36TB Micron 9300 Pro
>
> For production this will be extended to all 8 nodes if it shows
> promising results.
>
> - Ceph was installed with cephadm.
> - MDS and ODS are located on the same nodes.
> - Mostly using stock config
>
> - Network performance tested with iperf3 seems fine, 26Gbits/s with
> -P4 on single port (details below).
> Close to 200Gbits with 10 parallel instances and servers.
>
> When testing a mounted CephFS on the working nodes in various
> configurations I only got <50MB/s for fuse mount and <270MB/s for
> kernel mounts. (dd command and output attached below)
> In addition ceph dashboard and our graphana monitoring reports packet
> loss on all relevant interfaces during load. Which does not occur
> during the normal iperf load tests or rsync/scp file transfer.
>
> Rados Bench shows performance around 2000MB/s which is not max
> performance of the SSDs but fine for us (details below).
>
>
> Why is the filesystem so slow compared to the individual components?
>
> Cheers
> Dominik
>
>
>
>
> Test details:
>
>
------------------------------------------------------------------------------------------------------
>
>
> Some tests done on working nodes:
>
> Ceph mounted with ceph-fuse
>
> root@ml2ran10:/mnt/cephfs/backup# dd if=/dev/zero of=testfile bs=1M
> count=4096 oflag=direct
> 4096+0 records in
> 4096+0 records out
> 4294967296 bytes (4,3 GB, 4,0 GiB) copied, 88,2933 s, 48,6 MB/s
>
>
> Ceph mounted with kernel driver:
>
> root@ml2ran06:/mnt/cephfs/backup# dd if=/dev/zero of=testfile bs=1M
> count=4096 oflag=direct
> 4096+0 records in
> 4096+0 records out
> 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.0989 s, 267 MB/s
>
>
> Storage Node
>
> With fuse
>
> root@ml2rsn05:/mnt/ml2r_storage/backup# dd if=/dev/zero of=testfile
> bs=1M count=4096 oflag=direct
> 4096+0 records in
> 4096+0 records out
> 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 53.9977 s, 79.5 MB/s
>
> Kernel mount:
>
> dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct
> 4096+0 records in
> 4096+0 records out
> 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.6726 s, 243 MB/s
>
> _______________________________________________________
>
> Iperf3
>
> iperf3 --zerocopy -n 10240M -P4 -c ml2ran08s0 -p 4701 -i 15 -b
> 200000000000
> Connecting to host ml2ran08s0, port 4701
> [ 5] local 129.217.31.180 port 43958 connected to 129.217.31.218 port
> 4701
> [ 7] local 129.217.31.180 port 43960 connected to 129.217.31.218 port
> 4701
> [ 9] local 129.217.31.180 port 43962 connected to 129.217.31.218 port
> 4701
> [ 11] local 129.217.31.180 port 43964 connected to 129.217.31.218 port
> 4701
> [ ID] Interval Transfer Bitrate Retr Cwnd
> [ 5] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 632 KBytes
> [ 7] 0.00-3.21 sec 2.50 GBytes 6.70 Gbits/sec 0 522 KBytes
> [ 9] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 612 KBytes
> [ 11] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 430 KBytes
> [SUM] 0.00-3.21 sec 10.0 GBytes 26.8 Gbits/sec 0
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bitrate Retr
> [ 5] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender
> [ 5] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec
> receiver
> [ 7] 0.00-3.21 sec 2.50 GBytes 6.70 Gbits/sec 0 sender
> [ 7] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec
> receiver
> [ 9] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender
> [ 9] 0.00-3.21 sec 2.49 GBytes 6.67 Gbits/sec
> receiver
> [ 11] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender
> [ 11] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec
> receiver
> [SUM] 0.00-3.21 sec 10.0 GBytes 26.8 Gbits/sec 0 sender
> [SUM] 0.00-3.21 sec 9.98 GBytes 26.7 Gbits/sec
> receiver
>
>
>
> _________________________________________________________________
>
> Rados Bench on storage node:
>
> # rados bench -p testbench 10 write --no-cleanup
> hints = 1
> Maintaining 16 concurrent writes of 4194304 bytes to objects of size
> 4194304 for up to 10 seconds or 0 objects
> Object prefix: benchmark_data_ml2rsn05_2829244
> sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
> lat(s)
> 0 0 0 0 0 0 - 0
> 1 16 747 731 2923.84 2924 0.0178757 0.0216262
> 2 16 1506 1490 2979.71 3036 0.0308664 0.0213685
> 3 16 2267 2251 3000.99 3044 0.0259053 0.0212556
> 4 16 3058 3042 3041.62 3164 0.0227621 0.0209792
> 5 16 3850 3834 3066.8 3168 0.0130519 0.0208148
> 6 16 4625 4609 3072.26 3100 0.151371 0.0207904
> 7 16 5381 5365 3065.28 3024 0.0300368 0.0208345
> 8 16 6172 6156 3077.57 3164 0.0197728 0.0207714
> 9 16 6971 6955 3090.67 3196 0.0142751 0.0206786
> 10 14 7772 7758 3102.76 3212 0.0181034 0.020605
> Total time run: 10.0179
> Total writes made: 7772
> Write size: 4194304
> Object size: 4194304
> Bandwidth (MB/sec): 3103.23
> Stddev Bandwidth: 93.3676
> Max bandwidth (MB/sec): 3212
> Min bandwidth (MB/sec): 2924
> Average IOPS: 775
> Stddev IOPS: 23.3419
> Max IOPS: 803
> Min IOPS: 731
> Average Latency(s): 0.020598
> Stddev Latency(s): 0.00731743
> Max latency(s): 0.151371
> Min latency(s): 0.00966991
>
>
>
> # rados bench -p testbench 10 seq
> hints = 1
> sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
> lat(s)
> 0 0 0 0 0 0 - 0
> 1 15 657 642 2567.32 2568 0.011104 0.022631
> 2 15 1244 1229 2456.9 2348 0.0115248 0.019485
> 3 16 1499 1483 1976.64 1016 0.00722983
> 0.0177887
> 4 15 1922 1907 1906.16 1696 0.0142242 0.0327382
> 5 15 2593 2578 2061.6 2684 0.011758 0.0301774
> 6 16 3142 3126 2083.23 2192 0.00915926
> 0.027478
> 7 16 3276 3260 1862.23 536 0.00824714
> 0.0267449
> 8 16 3606 3590 1794.43 1320 0.0118938 0.0350541
> 9 16 4293 4277 1900.32 2748 0.0301886 0.0330604
> 10 14 5003 4989 1995.04 2848 0.0389717 0.0314977
> Total time run: 10.0227
> Total reads made: 5003
> Read size: 4194304
> Object size: 4194304
> Bandwidth (MB/sec): 1996.67
> Average IOPS: 499
> Stddev IOPS: 202.3
> Max IOPS: 712
> Min IOPS: 134
> Average Latency(s): 0.0314843
> Max latency(s): 3.04463
> Min latency(s): 0.00551523
>
>
> # rados bench -p testbench 10 rand
> hints = 1
> sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
> lat(s)
> 0 15 15 0 0 0 - 0
> 1 15 680 665 2657.61 2660 0.00919807
> 0.0224833
> 2 15 1273 1258 2514.26 2372 0.00839656
> 0.0247125
> 3 16 1863 1847 2461.4 2356 0.00994467
> 0.0236565
> 4 16 2064 2048 2047.14 804 0.00809139
> 0.0223506
> 5 16 2064 2048 1637.79 0 - 0.0223506
> 6 16 2477 2461 1640.12 826 0.0286315 0.0383254
> 7 16 3102 3086 1762.89 2500 0.0267464 0.0349189
> 8 16 3513 3497 1748 1644 0.00890952
> 0.032269
> 9 16 3617 3601 1600 416 0.00626917
> 0.0316019
> 10 15 4014 3999 1599.18 1592 0.0461076 0.0393606
> Total time run: 10.0481
> Total reads made: 4014
> Read size: 4194304
> Object size: 4194304
> Bandwidth (MB/sec): 1597.91
> Average IOPS: 399
> Stddev IOPS: 239.089
> Max IOPS: 665
> Min IOPS: 0
> Average Latency(s): 0.0394035
> Max latency(s): 3.00962
> Min latency(s): 0.00449537
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
23 Mar
23 Mar
7:24 a.m.
Hi Dominik,
RADOS bench will perform parallel IOs, which stresses the internal
configuration, but it would not be the speed of an individual client. Ceph
is inherently designed for fairness, due to the pseudo random distribution
of data, and the sharded storage design. Kernel mounts are going to be
fastest, and you can play with caching parameters, and things like
readahead.
When reading, you are reading from one OSD at a time, when writing you are
writing to all redundant OSDs (three if you use 3x replication, or however
many with an erasure coded setup). Writing is acknowledged only when all
participating OSDs have completed.hardened their writes, so you will have
network latency + OS overhead + ceph overhead + drive latency for each
write.
We have used tricks, such as parallelizing IO across multiple RBD images,
and increasing queue depth, but that's not cephfs, rather ZFS or XFS on top
of RBD. With proper block alignment, we have seen reasonable performance
from such setups.
For your network links, are you using 802.3ad aggregation and have your MTU
correctly set across the board - client, OSD nods, MONs? You will want the
MTU to be ideally the same (1500, 9216, 9000 etc.) across the entire
cluster. Check your hashing algorithm (we use layer 2+3 for most setups).
I would also focus on benchmarking what you would use this cluster for, if
there's one thing I learned from the storage industry is that there are
lies, damn lies, and benchmarks. If your workload goes to 5000 IOPS max,
you do not need a million IOPS. If you need good latency response, buy the
best NVMe drives possible for your use case, because latency will always go
all the way to the drive itself.
Hope this helps, and others can likely address cephfs aspects for you.
--
Alex Gorbachev
https://alextelescope.blogspot.com
On Wed, Mar 22, 2023 at 10:16 AM Dominik Baack <
dominik.baack(a)cs.uni-dortmund.de> wrote:
Hi,
we are currently testing out ways to increase Ceph performance because
what we experience so far is very close to unusable.
For the test cluster we utilizing 4 nodes with the following hardware data:
Dual 200GBe Mellanox Ethernet
2x EPYC Rome 7302
16x 32GB 3200MHz ECC
9x 15.36TB Micron 9300 Pro
For production this will be extended to all 8 nodes if it shows
promising results.
- Ceph was installed with cephadm.
- MDS and ODS are located on the same nodes.
- Mostly using stock config
- Network performance tested with iperf3 seems fine, 26Gbits/s with -P4
on single port (details below).
Close to 200Gbits with 10 parallel instances and servers.
When testing a mounted CephFS on the working nodes in various
configurations I only got <50MB/s for fuse mount and <270MB/s for kernel
mounts. (dd command and output attached below)
In addition ceph dashboard and our graphana monitoring reports packet
loss on all relevant interfaces during load. Which does not occur during
the normal iperf load tests or rsync/scp file transfer.
Rados Bench shows performance around 2000MB/s which is not max
performance of the SSDs but fine for us (details below).
Why is the filesystem so slow compared to the individual components?
Cheers
Dominik
Test details:
------------------------------------------------------------------------------------------------------
Some tests done on working nodes:
Ceph mounted with ceph-fuse
root@ml2ran10:/mnt/cephfs/backup# dd if=/dev/zero of=testfile bs=1M
count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4,3 GB, 4,0 GiB) copied, 88,2933 s, 48,6 MB/s
Ceph mounted with kernel driver:
root@ml2ran06:/mnt/cephfs/backup# dd if=/dev/zero of=testfile bs=1M
count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.0989 s, 267 MB/s
Storage Node
With fuse
root@ml2rsn05:/mnt/ml2r_storage/backup# dd if=/dev/zero of=testfile
bs=1M count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 53.9977 s, 79.5 MB/s
Kernel mount:
dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.6726 s, 243 MB/s
_______________________________________________________
Iperf3
iperf3 --zerocopy -n 10240M -P4 -c ml2ran08s0 -p 4701 -i 15 -b
200000000000
Connecting to host ml2ran08s0, port 4701
[ 5] local 129.217.31.180 port 43958 connected to 129.217.31.218 port 4701
[ 7] local 129.217.31.180 port 43960 connected to 129.217.31.218 port 4701
[ 9] local 129.217.31.180 port 43962 connected to 129.217.31.218 port 4701
[ 11] local 129.217.31.180 port 43964 connected to 129.217.31.218 port 4701
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 632 KBytes
[ 7] 0.00-3.21 sec 2.50 GBytes 6.70 Gbits/sec 0 522 KBytes
[ 9] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 612 KBytes
[ 11] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 430 KBytes
[SUM] 0.00-3.21 sec 10.0 GBytes 26.8 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender
[ 5] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec
receiver
[ 7] 0.00-3.21 sec 2.50 GBytes 6.70 Gbits/sec 0 sender
[ 7] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec
receiver
[ 9] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender
[ 9] 0.00-3.21 sec 2.49 GBytes 6.67 Gbits/sec
receiver
[ 11] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender
[ 11] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec
receiver
[SUM] 0.00-3.21 sec 10.0 GBytes 26.8 Gbits/sec 0 sender
[SUM] 0.00-3.21 sec 9.98 GBytes 26.7 Gbits/sec
receiver
_________________________________________________________________
Rados Bench on storage node:
# rados bench -p testbench 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size
4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_ml2rsn05_2829244
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 0 0 0 0 0 - 0
1 16 747 731 2923.84 2924 0.0178757
0.0216262
2 16 1506 1490 2979.71 3036 0.0308664
0.0213685
3 16 2267 2251 3000.99 3044 0.0259053
0.0212556
4 16 3058 3042 3041.62 3164 0.0227621
0.0209792
5 16 3850 3834 3066.8 3168 0.0130519
0.0208148
6 16 4625 4609 3072.26 3100 0.151371 0.0207904
7 16 5381 5365 3065.28 3024 0.0300368
0.0208345
8 16 6172 6156 3077.57 3164 0.0197728
0.0207714
9 16 6971 6955 3090.67 3196 0.0142751
0.0206786
10 14 7772 7758 3102.76 3212 0.0181034
0.020605
Total time run: 10.0179
Total writes made: 7772
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 3103.23
Stddev Bandwidth: 93.3676
Max bandwidth (MB/sec): 3212
Min bandwidth (MB/sec): 2924
Average IOPS: 775
Stddev IOPS: 23.3419
Max IOPS: 803
Min IOPS: 731
Average Latency(s): 0.020598
Stddev Latency(s): 0.00731743
Max latency(s): 0.151371
Min latency(s): 0.00966991
# rados bench -p testbench 10 seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 0 0 0 0 0 - 0
1 15 657 642 2567.32 2568 0.011104 0.022631
2 15 1244 1229 2456.9 2348 0.0115248
0.019485
3 16 1499 1483 1976.64 1016 0.00722983
0.0177887
4 15 1922 1907 1906.16 1696 0.0142242
0.0327382
5 15 2593 2578 2061.6 2684 0.011758 0.0301774
6 16 3142 3126 2083.23 2192 0.00915926
0.027478
7 16 3276 3260 1862.23 536 0.00824714
0.0267449
8 16 3606 3590 1794.43 1320 0.0118938
0.0350541
9 16 4293 4277 1900.32 2748 0.0301886
0.0330604
10 14 5003 4989 1995.04 2848 0.0389717
0.0314977
Total time run: 10.0227
Total reads made: 5003
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 1996.67
Average IOPS: 499
Stddev IOPS: 202.3
Max IOPS: 712
Min IOPS: 134
Average Latency(s): 0.0314843
Max latency(s): 3.04463
Min latency(s): 0.00551523
# rados bench -p testbench 10 rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 15 15 0 0 0 - 0
1 15 680 665 2657.61 2660 0.00919807
0.0224833
2 15 1273 1258 2514.26 2372 0.00839656
0.0247125
3 16 1863 1847 2461.4 2356 0.00994467
0.0236565
4 16 2064 2048 2047.14 804 0.00809139
0.0223506
5 16 2064 2048 1637.79 0 - 0.0223506
6 16 2477 2461 1640.12 826 0.0286315
0.0383254
7 16 3102 3086 1762.89 2500 0.0267464
0.0349189
8 16 3513 3497 1748 1644 0.00890952
0.032269
9 16 3617 3601 1600 416 0.00626917
0.0316019
10 15 4014 3999 1599.18 1592 0.0461076
0.0393606
Total time run: 10.0481
Total reads made: 4014
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 1597.91
Average IOPS: 399
Stddev IOPS: 239.089
Max IOPS: 665
Min IOPS: 0
Average Latency(s): 0.0394035
Max latency(s): 3.00962
Min latency(s): 0.00449537
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
420
days inactive
422
days old
2 comments
3 participants
participants (3)
-
Alex Gorbachev -
Dominik Baack -
Joachim Kraftmayer