hello I got a huge difference of read/write performance on my ceph cluster when I use rbd. rbd bench reach the limit of my cluster (1gbps network) when the performance in mapped rbd is very low. 30MB/s. ceph version : ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable) kernel version : 4.19.0-5-amd64 ceph -s cluster: id: fdf6eba0-cde9-401c-8901-751eb1e8b7d2 health: HEALTH_OK services: mon: 3 daemons, quorum gra3-k8s001,sbg2-k8s001,rbx4-k8s001 (age 41h) mgr: sbg2-k8s001(active, since 46h), standbys: gra3-k8s001, rbx4- k8s001 osd: 10 osds: 10 up (since 41h), 10 in (since 41h) rgw: 3 daemons active (gra3-k8s001, rbx4-k8s001, sbg2-k8s001) data: pools: 8 pools, 240 pgs objects: 109.14k objects, 229 GiB usage: 694 GiB used, 3.0 TiB / 3.7 TiB avail pgs: 240 active+clean here the bench rbd create --size=10G kube/bench rbd feature disable kube/bench object-map fast-diff deep-flatten rbd -p kube bench kube/bench --io-type write --io-size 8192 --io- threads 256 --io-total 10G --io-pattern seq elapsed: 110 ops: 1310720 ops/sec: 11884.39 bytes/sec: 97356926.64 this reach the 1gbps limit of my network. Looks good. let's test random write rbd -p kube bench kube/bench --io-type write --io-size 8192 --io- threads 256 --io-total 10G --io-pattern rand elapsed: 125 ops: 1310720 ops/sec: 10416.31 bytes/sec: 85330446.58 this reach the 1gbps limit of my network. Looks good. rbd -p kube bench kube/bench --io-type read --io-size 8192 --io-threads 256 --io-total 10G --io-pattern seq elapsed: 40 ops: 1310720 ops/sec: 32712.03 bytes/sec: 267976974.35 this is great perf ! let's try rand rbd -p kube bench kube/bench --io-type read --io-size 8192 --io-threads 256 --io-total 10G --io-pattern rand elapsed: 29 ops: 1310720 ops/sec: 43859.78 bytes/sec: 359299337.92 this is great perf ! But when I mount the image : rbd map bench --pool kube --name client.admin /sbin/mkfs.ext4 /dev/rbd/kube/bench mount /dev/rbd/kube/bench /mnt/ cd /mnt/ dd if=/dev/zero of=test bs=8192k count=100 oflag=direct 838860800 bytes (839 MB, 800 MiB) copied, 24.6185 s, 34.1 MB/s 34.1MB/s vs 85MB/s ....
On Wed, Aug 14, 2019 at 2:38 PM Olivier AUDRY <olivier@nmlq.fr> wrote:
let's test random write rbd -p kube bench kube/bench --io-type write --io-size 8192 --io-threads 256 --io-total 10G --io-pattern rand elapsed: 125 ops: 1310720 ops/sec: 10416.31 bytes/sec: 85330446.58
dd if=/dev/zero of=test bs=8192k count=100 oflag=direct 838860800 bytes (839 MB, 800 MiB) copied, 24.6185 s, 34.1 MB/s
34.1MB/s vs 85MB/s ....
34 apples vs. 85 oranges You are comparing 256 threads with a huge queue depth vs a single thread with a normal queue depth. Use fio on the mounted rbd to get better control over what it's doing Paul
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Aug 14, 2019 at 2:49 PM Paul Emmerich <paul.emmerich@croit.io> wrote:
On Wed, Aug 14, 2019 at 2:38 PM Olivier AUDRY <olivier@nmlq.fr> wrote:
let's test random write rbd -p kube bench kube/bench --io-type write --io-size 8192 --io-threads 256 --io-total 10G --io-pattern rand elapsed: 125 ops: 1310720 ops/sec: 10416.31 bytes/sec: 85330446.58
dd if=/dev/zero of=test bs=8192k count=100 oflag=direct 838860800 bytes (839 MB, 800 MiB) copied, 24.6185 s, 34.1 MB/s
34.1MB/s vs 85MB/s ....
34 apples vs. 85 oranges
You are comparing 256 threads with a huge queue depth vs a single thread with a normal queue depth. Use fio on the mounted rbd to get better control over what it's doing
When you said mounted, did you mean mapped or "a filesystem mounted on top of a mapped rbd"? There is no filesystem in "rbd bench" tests, so fio should be used on a raw block device. It still won't be completely apples to apples because in "rbd bench" or fio's rbd engine (--ioengine=rbd) case there is no block layer either, but it is closer... Thanks, Ilya
hello I mean a filesystem mounted on top of a mapped rbd rbd create --size=10G kube/benchrbd feature disable kube/bench object- map fast-diff deep-flattenrbd map bench --pool kube --name client.admin/sbin/mkfs.ext4 /dev/rbd/kube/bench mount /dev/rbd/kube/bench /mnt/cd /mnt/ about the bench I did. I try to get apples in both side. (I hope) : block size : 4kthread : 1size of data : 1G Writes are great. rbd -p kube bench kube/bench --io-type write --io-threads 1 --io-total 1G --io-pattern seq elapsed: 12 ops: 262144 ops/sec: 20758.70 bytes/sec: 85027625.70 rbd -p kube bench kube/bench --io-type write --io-threads 1 --io-total 10G --io-pattern rand elapsed: 14 ops: 262144 ops/sec: 17818.16 bytes/sec: 72983201.32 Reads are very very slow :rbd -p kube bench kube/bench --io-type read --io-threads 1 --io-total 1G --io-pattern randelapsed: 445 ops: 81216 ops/sec: 182.37 bytes/sec: 747006.15 rbd -p kube bench kube/bench --io-type read --io-threads 1 --io-total 1G --io-pattern seqelapsed: 14 ops: 14153 ops/sec: 957.57 bytes/sec: 3922192.15 Perhaps I'm hitting this 'issue' : http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028878.html For the record : I got an old cluster with vm in ceph 10.2.11. With a dd bench I reach the cluster limitation. dd if=/dev/zero of=test bs=4M count=250 oflag=direct1048576000 bytes (1.0 GB) copied, 11.5469 s, 90.8 MB/s and pgbench gives me 200 transaction per second On the new cluster with containers running on fs on top of a mapped rbd and ceph nautilus I got :dd if=/dev/zero of=test bs=4M count=250 oflag=direct1048576000 bytes (1.0 GB, 1000 MiB) copied, 27.0351 s, 38.8 MB/s and pgbench gives me 10 transactions per second. something it not ok somewhere :) oau Le mercredi 14 août 2019 à 15:56 +0200, Ilya Dryomov a écrit :
On Wed, Aug 14, 2019 at 2:49 PM Paul Emmerich <paul.emmerich@croit.io
wrote: On Wed, Aug 14, 2019 at 2:38 PM Olivier AUDRY <olivier@nmlq.fr> wrote:
let's test random write rbd -p kube bench kube/bench --io-type write --io-size 8192 --io- threads 256 --io-total 10G --io-pattern rand elapsed: 125 ops: 1310720 ops/sec: 10416.31 bytes/sec: 85330446.58
dd if=/dev/zero of=test bs=8192k count=100 oflag=direct 838860800 bytes (839 MB, 800 MiB) copied, 24.6185 s, 34.1 MB/s
34.1MB/s vs 85MB/s ....
34 apples vs. 85 oranges
You are comparing 256 threads with a huge queue depth vs a single thread with a normal queue depth. Use fio on the mounted rbd to get better control over what it's doing
When you said mounted, did you mean mapped or "a filesystem mounted on top of a mapped rbd"?
There is no filesystem in "rbd bench" tests, so fio should be used on a raw block device. It still won't be completely apples to apples because in "rbd bench" or fio's rbd engine (--ioengine=rbd) case there is no block layer either, but it is closer...
Thanks,
Ilya
rbd -p kube bench kube/bench --io-type write --io-threads 1 --io-total 10G --io-pattern rand elapsed: 14 ops: 262144 ops/sec: 17818.16 bytes/sec: 72983201.32
It's a totally unreal number. Something is wrong with the test. Test it with `fio` please: fio -ioengine=rbd -name=test -bs=4k -iodepth=1 -rw=randwrite -runtime=60 -pool=kube -rbdname=bench
Reads are very very slow: elapsed: 445 ops: 81216 ops/sec: 182.37 bytes/sec: 747006.15 elapsed: 14 ops: 14153 ops/sec: 957.57 bytes/sec: 3922192.15
This is closer to the reality. -- With best regards, Vitaliy Filippov
hello here the result : fio --ioengine=rbd --name=test --bs=4k --iodepth=1 --rw=randwrite -- runtime=60 -pool=kube -rbdname=bench test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.12 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][w=248KiB/s][w=62 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=1903256: Fri Aug 16 13:22:59 2019 write: IOPS=58, BW=232KiB/s (238kB/s)(13.6MiB/60011msec); 0 zone resets slat (usec): min=9, max=351, avg=52.82, stdev=23.02 clat (usec): min=1264, max=96970, avg=17156.70, stdev=6713.88 lat (usec): min=1276, max=97050, avg=17209.52, stdev=6715.06 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3884], 10.00th=[11863], 20.00th=[13304], | 30.00th=[13960], 40.00th=[14484], 50.00th=[15008], 60.00th=[20579], | 70.00th=[22152], 80.00th=[23987], 90.00th=[25297], 95.00th=[25822], | 99.00th=[26346], 99.50th=[27395], 99.90th=[71828], 99.95th=[82314], | 99.99th=[96994] bw ( KiB/s): min= 104, max= 272, per=100.00%, avg=232.17, stdev=19.55, samples=120 iops : min= 26, max= 68, avg=57.97, stdev= 4.88, samples=120 lat (msec) : 2=0.06%, 4=5.51%, 10=3.41%, 20=50.22%, 50=40.69% lat (msec) : 100=0.11% cpu : usr=0.44%, sys=0.27%, ctx=3489, majf=0, minf=3582 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=0,3485,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): WRITE: bw=232KiB/s (238kB/s), 232KiB/s-232KiB/s (238kB/s-238kB/s), io=13.6MiB (14.3MB), run=60011-60011msec Disk stats (read/write): md2: ios=3/4611, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=3/8532, aggrmerge=0/2364, aggrticks=0/1346, aggrin_queue=52504, aggrutil=88.03% nvme1n1: ios=3/8529, merge=0/2295, ticks=1/1347, in_queue=52932, util=88.03% nvme0n1: ios=3/8535, merge=1/2434, ticks=0/1346, in_queue=52076, util=86.52% for you information my disk setup is : 2 500G nvme disk with : 10G raid 1 partition with the OS 160G raid0 partition for local docker data 387.00g partition on each disk for the osd. I got 5 physical devices with 12 core cpu and 32go RAM. Ceph is running on dedicated 1Gbps network. I got quite the same hardware setup with ceph 10.2.11 and I got much better performance. oau lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT rbd0 252:0 0 1G 0 disk /var/lib/kubelet/pods/db24eb3b-650c-42f3-bdf0- 92ea0eaf37d8/volumes/kubernetes.io~csi/pvc-e70f3d74-c7bd-4652-983b- de3874d36117/mou rbd1 252:16 0 1G 0 disk /var/lib/kubelet/pods/a1b0b156-0bcc-4d0c-b2f8- 26a74337baed/volumes/kubernetes.io~csi/pvc-14460f8a-da5a-44e3-a033- a3bf5054f967/mou rbd2 252:32 0 8G 0 disk /var/lib/kubelet/pods/0a5e3745-c6c2-49a9-971a- 3ddac59af66c/volumes/kubernetes.io~csi/pvc-b8907f44-58d6-4599-b189- fafe65daed09/mou rbd3 252:48 0 50G 0 disk /var/lib/kubelet/pods/84df04af-a7dd-4035-a9a2- 22d3d315fa60/volumes/kubernetes.io~csi/pvc-faf5cefa-ecce-450a-9f5c- 42e7ca7d7fc2/mou rbd4 252:64 0 1G 0 disk /var/lib/kubelet/pods/872a3d7d-63d5-4567-86f6- bedab3fe0ad3/volumes/kubernetes.io~csi/pvc-83ea4013-b936-469b-b20f- 7e703db2c871/mou nvme0n1 259:0 0 477G 0 disk ├─nvme0n1p1 259:1 0 511M 0 part /boot/efi ├─nvme0n1p2 259:2 0 9.8G 0 part │ └─md2 9:2 0 9.8G 0 raid1 / ├─nvme0n1p3 259:3 0 79.5G 0 part │ └─md3 9:3 0 158.9G 0 raid0 │ └─datavg-dockerlv 253:0 0 30G 0 lvm /var/lib/docker └─nvme0n1p4 259:8 0 387.2G 0 part └─ceph--52ce0eb9--9e69--4f29--8b87--9ab3fbb5df3e-osd--block --c26bdb06--2325--4fcd--9b8c--77a93ab46de0 253:1 0 387G 0 lvm nvme1n1 259:4 0 477G 0 disk ├─nvme1n1p1 259:5 0 511M 0 part ├─nvme1n1p2 259:6 0 9.8G 0 part │ └─md2 9:2 0 9.8G 0 raid1 / ├─nvme1n1p3 259:7 0 79.5G 0 part │ └─md3 9:3 0 158.9G 0 raid0 │ └─datavg-dockerlv 253:0 0 30G 0 lvm /var/lib/docker └─nvme1n1p4 259:9 0 387.2G 0 part └─ceph--4ef4eb77--4b23--4ffd--b7d2--0fd2cfa5e568-osd--block --28435fba--91c7--4424--99f2--ea86709a87ca 253:2 0 387G 0 lvm Le vendredi 16 août 2019 à 01:16 +0300, Vitaliy Filippov a écrit :
rbd -p kube bench kube/bench --io-type write --io-threads 1 --io- total 10G --io-pattern rand elapsed: 14 ops: 262144 ops/sec: 17818.16 bytes/sec: 72983201.32
It's a totally unreal number. Something is wrong with the test.
Test it with `fio` please:
fio -ioengine=rbd -name=test -bs=4k -iodepth=1 -rw=randwrite -runtime=60 -pool=kube -rbdname=bench
Reads are very very slow: elapsed: 445 ops: 81216 ops/sec: 182.37 bytes/sec: 747006.15 elapsed: 14 ops: 14153 ops/sec: 957.57 bytes/sec: 3922192.15
This is closer to the reality.
hello just for the record the nvme disk are pretty fast. dd if=/dev/zero of=test bs=8192k count=100 oflag=direct 100+0 records in 100+0 records out 838860800 bytes (839 MB, 800 MiB) copied, 0.49474 s, 1.7 GB/s oau Le vendredi 16 août 2019 à 13:31 +0200, Olivier AUDRY a écrit :
hello
here the result :
fio --ioengine=rbd --name=test --bs=4k --iodepth=1 --rw=randwrite -- runtime=60 -pool=kube -rbdname=bench test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.12 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][w=248KiB/s][w=62 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=1903256: Fri Aug 16 13:22:59 2019 write: IOPS=58, BW=232KiB/s (238kB/s)(13.6MiB/60011msec); 0 zone resets slat (usec): min=9, max=351, avg=52.82, stdev=23.02 clat (usec): min=1264, max=96970, avg=17156.70, stdev=6713.88 lat (usec): min=1276, max=97050, avg=17209.52, stdev=6715.06 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3884], 10.00th=[11863], 20.00th=[13304], | 30.00th=[13960], 40.00th=[14484], 50.00th=[15008], 60.00th=[20579], | 70.00th=[22152], 80.00th=[23987], 90.00th=[25297], 95.00th=[25822], | 99.00th=[26346], 99.50th=[27395], 99.90th=[71828], 99.95th=[82314], | 99.99th=[96994] bw ( KiB/s): min= 104, max= 272, per=100.00%, avg=232.17, stdev=19.55, samples=120 iops : min= 26, max= 68, avg=57.97, stdev= 4.88, samples=120 lat (msec) : 2=0.06%, 4=5.51%, 10=3.41%, 20=50.22%, 50=40.69% lat (msec) : 100=0.11% cpu : usr=0.44%, sys=0.27%, ctx=3489, majf=0, minf=3582 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=0,3485,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): WRITE: bw=232KiB/s (238kB/s), 232KiB/s-232KiB/s (238kB/s-238kB/s), io=13.6MiB (14.3MB), run=60011-60011msec
Disk stats (read/write): md2: ios=3/4611, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=3/8532, aggrmerge=0/2364, aggrticks=0/1346, aggrin_queue=52504, aggrutil=88.03% nvme1n1: ios=3/8529, merge=0/2295, ticks=1/1347, in_queue=52932, util=88.03% nvme0n1: ios=3/8535, merge=1/2434, ticks=0/1346, in_queue=52076, util=86.52%
for you information my disk setup is :
2 500G nvme disk with :
10G raid 1 partition with the OS 160G raid0 partition for local docker data 387.00g partition on each disk for the osd.
I got 5 physical devices with 12 core cpu and 32go RAM. Ceph is running on dedicated 1Gbps network.
I got quite the same hardware setup with ceph 10.2.11 and I got much better performance.
oau
lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT rbd0 252:0 0 1G 0 disk /var/lib/kubelet/pods/db24eb3b-650c-42f3-bdf0- 92ea0eaf37d8/volumes/kubernetes.io~csi/pvc-e70f3d74-c7bd-4652-983b- de3874d36117/mou rbd1 252:16 0 1G 0 disk /var/lib/kubelet/pods/a1b0b156-0bcc-4d0c-b2f8- 26a74337baed/volumes/kubernetes.io~csi/pvc-14460f8a-da5a-44e3-a033- a3bf5054f967/mou rbd2 252:32 0 8G 0 disk /var/lib/kubelet/pods/0a5e3745-c6c2-49a9-971a- 3ddac59af66c/volumes/kubernetes.io~csi/pvc-b8907f44-58d6-4599-b189- fafe65daed09/mou rbd3 252:48 0 50G 0 disk /var/lib/kubelet/pods/84df04af-a7dd-4035-a9a2- 22d3d315fa60/volumes/kubernetes.io~csi/pvc-faf5cefa-ecce-450a-9f5c- 42e7ca7d7fc2/mou rbd4 252:64 0 1G 0 disk /var/lib/kubelet/pods/872a3d7d-63d5-4567-86f6- bedab3fe0ad3/volumes/kubernetes.io~csi/pvc-83ea4013-b936-469b-b20f- 7e703db2c871/mou nvme0n1 259:0 0 477G 0 disk ├─nvme0n1p1 259:1 0 511M 0 part /boot/efi ├─nvme0n1p2 259:2 0 9.8G 0 part │ └─md2 9:2 0 9.8G 0 raid1 / ├─nvme0n1p3 259:3 0 79.5G 0 part │ └─md3 9:3 0 158.9G 0 raid0 │ └─datavg-dockerlv 253:0 0 30G 0 lvm /var/lib/docker └─nvme0n1p4 259:8 0 387.2G 0 part └─ceph--52ce0eb9--9e69--4f29--8b87--9ab3fbb5df3e-osd--block --c26bdb06--2325--4fcd--9b8c--77a93ab46de0 253:1 0 387G 0 lvm nvme1n1 259:4 0 477G 0 disk ├─nvme1n1p1 259:5 0 511M 0 part ├─nvme1n1p2 259:6 0 9.8G 0 part │ └─md2 9:2 0 9.8G 0 raid1 / ├─nvme1n1p3 259:7 0 79.5G 0 part │ └─md3 9:3 0 158.9G 0 raid0 │ └─datavg-dockerlv 253:0 0 30G 0 lvm /var/lib/docker └─nvme1n1p4 259:9 0 387.2G 0 part └─ceph--4ef4eb77--4b23--4ffd--b7d2--0fd2cfa5e568-osd--block --28435fba--91c7--4424--99f2--ea86709a87ca 253:2 0 387G 0 lvm
Le vendredi 16 août 2019 à 01:16 +0300, Vitaliy Filippov a écrit :
rbd -p kube bench kube/bench --io-type write --io-threads 1 --io- total 10G --io-pattern rand elapsed: 14 ops: 262144 ops/sec: 17818.16 bytes/sec: 72983201.32
It's a totally unreal number. Something is wrong with the test.
Test it with `fio` please:
fio -ioengine=rbd -name=test -bs=4k -iodepth=1 -rw=randwrite -runtime=60 -pool=kube -rbdname=bench
Reads are very very slow: elapsed: 445 ops: 81216 ops/sec: 182.37 bytes/sec: 747006.15 elapsed: 14 ops: 14153 ops/sec: 957.57 bytes/sec: 3922192.15
This is closer to the reality.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Now to go for "apples to apples" either run fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randwrite -runtime=60 -filename=/dev/nvmeXXXXXXXXX to compare with the single-threaded RBD random write result (the test is destructive, so use a separate partition without data) ...Or run fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -pool=kube -rbdname=bench to compare with your dd's linear write result. 58 single-threaded random iops for NVMes is pretty sad either way. Are you NVMe's server ones? Do they have capacitors? :) in the response to a likely question about what they are I'll just post my link here again :) https://yourcmc.ru/wiki/Ceph_performance -- Vitaliy Filippov
hello here on the nvme partition directly - libaio randwrite /dev/nvme1n1p4 => WRITE: bw=12.1MiB/s (12.7MB/s), 12.1MiB/s-12.1MiB/s (12.7MB/s-12.7MB/s), io=728MiB (763MB), run=60001- 60001msec - libaio randread /dev/nvme1n1p4 => READ: bw=35.6MiB/s (37.3MB/s), 35.6MiB/s-35.6MiB/s (37.3MB/s-37.3MB/s), io=2134MiB (2237MB), run=60001-60001msec here on the rbd - rbd read : READ: bw=580MiB/s (608MB/s), 580MiB/s-580MiB/s (608MB/s- 608MB/s), io=10.0GiB (10.7GB), run=17668-17668msec (I want this perf ! :) ) - rbd write : WRITE: bw=90.9MiB/s (95.3MB/s), 90.9MiB/s-90.9MiB/s (95.3MB/s-95.3MB/s), io=5764MiB (6044MB), run=63404-63404msec (I want this perf ! :) ) here on the mapped rbd - libaio randwrite on mapped rbd : WRITE: bw=217KiB/s (223kB/s), 217KiB/s-217KiB/s (223kB/s-223kB/s), io=12.7MiB (13.4MB), run=60006- 60006msec - libaio randread on mapped rbd : READ: bw=589KiB/s (603kB/s), 589KiB/s-589KiB/s (603kB/s-603kB/s), io=34.5MiB (36.2MB), run=60005- 60005msec here on the mounted fs : rbd map bench --pool kube --name client.admin /sbin/mkfs.ext4 /dev/rbd/kube/bench mount /dev/rbd/kube/bench /mnt/ cd /mnt/ dd if=/dev/zero of=test bs=8192k count=100 oflag=direct 838860800 bytes (839 MB, 800 MiB) copied, 24.5338 s, 34.2 MB/s raw nvme performance does not looks very great .... but raw performance of the rbd are great Once I map it. The performance goes bad. When I use dd in the fs or fio on the device. what I don't understand is why the difference between raw rbd and mapped rbd are so important. The nvme disk looks to be : Sandisk Corp WD Black 2018/PC SN720 NVMe SSD Rand write : fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randwrite -runtime=60 -filename=/dev/nvme1n1p4 test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][w=12.3MiB/s][w=3158 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=177938: Fri Aug 16 17:18:18 2019 write: IOPS=3105, BW=12.1MiB/s (12.7MB/s)(728MiB/60001msec); 0 zone resets slat (nsec): min=1545, max=141856, avg=6784.48, stdev=6188.10 clat (nsec): min=688, max=3539.6k, avg=14665.22, stdev=13601.60 lat (usec): min=9, max=3549, avg=21.66, stdev=16.07 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 19], 90.00th=[ 26], 95.00th=[ 36], | 99.00th=[ 49], 99.50th=[ 52], 99.90th=[ 76], 99.95th=[ 85], | 99.99th=[ 135] bw ( KiB/s): min=10504, max=13232, per=100.00%, avg=12420.01, stdev=439.86, samples=119 iops : min= 2626, max= 3308, avg=3105.00, stdev=109.96, samples=119 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.02%, 4=0.09%, 10=40.60%, 20=43.62%, 50=14.92% lat (usec) : 100=0.73%, 250=0.02%, 500=0.01%, 750=0.01% lat (msec) : 4=0.01% fsync/fdatasync/sync_file_range: sync (nsec): min=12, max=22490, avg=185.20, stdev=255.75 sync percentiles (nsec): | 1.00th=[ 28], 5.00th=[ 35], 10.00th=[ 42], 20.00th=[ 59], | 30.00th=[ 82], 40.00th=[ 109], 50.00th=[ 137], 60.00th=[ 163], | 70.00th=[ 197], 80.00th=[ 253], 90.00th=[ 390], 95.00th=[ 572], | 99.00th=[ 804], 99.50th=[ 820], 99.90th=[ 1144], 99.95th=[ 1208], | 99.99th=[15552] cpu : usr=2.42%, sys=4.91%, ctx=546845, majf=0, minf=12 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=0,186313,0,186313 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): WRITE: bw=12.1MiB/s (12.7MB/s), 12.1MiB/s-12.1MiB/s (12.7MB/s- 12.7MB/s), io=728MiB (763MB), run=60001-60001msec Disk stats (read/write): nvme1n1: ios=0/375662, merge=0/1492, ticks=0/57478, in_queue=59100, util=97.85% RBD read ~# fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=read -runtime=60 -pool=kube -rbdname=bench test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=rbd, iodepth=32 fio-3.12 Starting 1 process Jobs: 1 (f=1): [R(1)][69.2%][r=1313MiB/s][r=328 IOPS][eta 00m:08s] test: (groupid=0, jobs=1): err= 0: pid=184386: Fri Aug 16 17:20:45 2019 read: IOPS=144, BW=580MiB/s (608MB/s)(10.0GiB/17668msec) slat (nsec): min=371, max=72009, avg=5416.87, stdev=3649.19 clat (msec): min=6, max=1438, avg=220.83, stdev=192.27 lat (msec): min=6, max=1438, avg=220.83, stdev=192.27 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 23], 10.00th=[ 25], 20.00th=[ 30], | 30.00th=[ 33], 40.00th=[ 47], 50.00th=[ 249], 60.00th=[ 288], | 70.00th=[ 326], 80.00th=[ 372], 90.00th=[ 447], 95.00th=[ 535], | 99.00th=[ 802], 99.50th=[ 844], 99.90th=[ 936], 99.95th=[ 1011], | 99.99th=[ 1435] bw ( KiB/s): min=253952, max=4628480, per=93.74%, avg=556353.83, stdev=851591.15, samples=35 iops : min= 62, max= 1130, avg=135.83, stdev=207.91, samples=35 lat (msec) : 10=0.08%, 20=2.03%, 50=38.40%, 100=2.15%, 250=7.62% lat (msec) : 500=43.48%, 750=4.73%, 1000=1.45% cpu : usr=0.19%, sys=0.24%, ctx=2563, majf=0, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=98.8%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, =64=0.0% issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs): READ: bw=580MiB/s (608MB/s), 580MiB/s-580MiB/s (608MB/s-608MB/s), io=10.0GiB (10.7GB), run=17668-17668msec Disk stats (read/write): md2: ios=1/1405, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1332/3191, aggrmerge=5911/5187, aggrticks=1031/1257, aggrin_queue=15480, aggrutil=91.95% nvme1n1: ios=0/1183, merge=0/450, ticks=0/141, in_queue=12452, util=69.94% nvme0n1: ios=2665/5199, merge=11822/9925, ticks=2062/2373, in_queue=18508, util=91.95% RBD write fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -pool=kube -rbdname=bench test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=rbd, iodepth=32 fio-3.12 Starting 1 process Jobs: 1 (f=1): [W(1)][55.3%][eta 00m:51s] test: (groupid=0, jobs=1): err= 0: pid=181899: Fri Aug 16 17:20:19 2019 write: IOPS=22, BW=90.9MiB/s (95.3MB/s)(5764MiB/63404msec); 0 zone resets slat (usec): min=366, max=4889, avg=802.71, stdev=414.92 clat (msec): min=189, max=7308, avg=1404.62, stdev=1278.48 lat (msec): min=189, max=7308, avg=1405.43, stdev=1278.47 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 376], 10.00th=[ 435], 20.00th=[ 550], | 30.00th=[ 676], 40.00th=[ 776], 50.00th=[ 877], 60.00th=[ 1083], | 70.00th=[ 1351], 80.00th=[ 1921], 90.00th=[ 3473], 95.00th=[ 4597], | 99.00th=[ 5470], 99.50th=[ 6007], 99.90th=[ 6745], 99.95th=[ 7282], | 99.99th=[ 7282] bw ( KiB/s): min= 8192, max=155648, per=100.00%, avg=95454.68, stdev=30757.07, samples=121 iops : min= 2, max= 38, avg=23.27, stdev= 7.52, samples=121 lat (msec) : 250=0.07%, 500=15.82%, 750=22.21%, 1000=18.74% cpu : usr=1.71%, sys=0.15%, ctx=597, majf=0, minf=45137 IO depths : 1=0.1%, 2=0.1%, 4=0.3%, 8=0.6%, 16=1.1%, 32=97.8%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, =64=0.0% issued rwts: total=0,1441,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs): WRITE: bw=90.9MiB/s (95.3MB/s), 90.9MiB/s-90.9MiB/s (95.3MB/s- 95.3MB/s), io=5764MiB (6044MB), run=63404-63404msec Disk stats (read/write): md2: ios=0/5119, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=30/16383, aggrmerge=20/28021, aggrticks=5/18949, aggrin_queue=63466, aggrutil=90.70% nvme1n1: ios=0/4230, merge=0/1595, ticks=0/1835, in_queue=42440, util=65.48% nvme0n1: ios=60/28536, merge=41/54447, ticks=10/36063, in_queue=84492, util=90.70% fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randread -runtime=60 -filename=/dev/nvme1n1p4 test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=36.3MiB/s][r=9286 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=208060: Fri Aug 16 17:33:01 2019 read: IOPS=9103, BW=35.6MiB/s (37.3MB/s)(2134MiB/60001msec) slat (nsec): min=1384, max=244751, avg=6077.86, stdev=5529.26 clat (usec): min=3, max=8311, avg=101.98, stdev=42.25 lat (usec): min=34, max=8341, avg=108.28, stdev=43.81 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 83], 10.00th=[ 85], 20.00th=[ 90], | 30.00th=[ 92], 40.00th=[ 94], 50.00th=[ 96], 60.00th=[ 100], | 70.00th=[ 104], 80.00th=[ 112], 90.00th=[ 121], 95.00th=[ 141], | 99.00th=[ 182], 99.50th=[ 198], 99.90th=[ 253], 99.95th=[ 297], | 99.99th=[ 2147] bw ( KiB/s): min=33088, max=40928, per=99.96%, avg=36400.00, stdev=1499.29, samples=119 iops : min= 8272, max=10232, avg=9099.98, stdev=374.82, samples=119 lat (usec) : 4=0.01%, 10=0.01%, 50=0.53%, 100=59.74%, 250=39.62% lat (usec) : 500=0.08%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=4.80%, sys=8.33%, ctx=546238, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=546228,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=35.6MiB/s (37.3MB/s), 35.6MiB/s-35.6MiB/s (37.3MB/s- 37.3MB/s), io=2134MiB (2237MB), run=60001-60001msec Disk stats (read/write): nvme1n1: ios=545111/3564, merge=0/1430, ticks=54388/943, in_queue=60924, util=100.00% Le vendredi 16 août 2019 à 18:06 +0300, vitalif@yourcmc.ru a écrit :
Now to go for "apples to apples" either run
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randwrite -runtime=60 -filename=/dev/nvmeXXXXXXXXX
to compare with the single-threaded RBD random write result (the test is destructive, so use a separate partition without data)
...Or run
fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -pool=kube -rbdname=bench
to compare with your dd's linear write result.
58 single-threaded random iops for NVMes is pretty sad either way. Are you NVMe's server ones? Do they have capacitors? :) in the response to a likely question about what they are I'll just post my link here again :) https://yourcmc.ru/wiki/Ceph_performance
Ok I read your link. My ssds are bad. They got capacitors ... I don't choose them. They come with the hardware I rent. Perhaps it will be better to switch to hdd. I cannot even but journal on them ... bad news :( Le vendredi 16 août 2019 à 17:37 +0200, Olivier AUDRY a écrit :
hello
here on the nvme partition directly
- libaio randwrite /dev/nvme1n1p4 => WRITE: bw=12.1MiB/s (12.7MB/s), 12.1MiB/s-12.1MiB/s (12.7MB/s-12.7MB/s), io=728MiB (763MB), run=60001-60001msec - libaio randread /dev/nvme1n1p4 => READ: bw=35.6MiB/s (37.3MB/s), 35.6MiB/s-35.6MiB/s (37.3MB/s-37.3MB/s), io=2134MiB (2237MB), run=60001-60001msec
here on the rbd
- rbd read : READ: bw=580MiB/s (608MB/s), 580MiB/s-580MiB/s (608MB/s- 608MB/s), io=10.0GiB (10.7GB), run=17668-17668msec (I want this perf ! :) ) - rbd write : WRITE: bw=90.9MiB/s (95.3MB/s), 90.9MiB/s-90.9MiB/s (95.3MB/s-95.3MB/s), io=5764MiB (6044MB), run=63404-63404msec (I want this perf ! :) )
here on the mapped rbd
- libaio randwrite on mapped rbd : WRITE: bw=217KiB/s (223kB/s), 217KiB/s-217KiB/s (223kB/s-223kB/s), io=12.7MiB (13.4MB), run=60006- 60006msec - libaio randread on mapped rbd : READ: bw=589KiB/s (603kB/s), 589KiB/s-589KiB/s (603kB/s-603kB/s), io=34.5MiB (36.2MB), run=60005- 60005msec
here on the mounted fs :
rbd map bench --pool kube --name client.admin /sbin/mkfs.ext4 /dev/rbd/kube/bench mount /dev/rbd/kube/bench /mnt/ cd /mnt/ dd if=/dev/zero of=test bs=8192k count=100 oflag=direct 838860800 bytes (839 MB, 800 MiB) copied, 24.5338 s, 34.2 MB/s
raw nvme performance does not looks very great .... but raw performance of the rbd are great Once I map it. The performance goes bad. When I use dd in the fs or fio on the device.
what I don't understand is why the difference between raw rbd and mapped rbd are so important.
The nvme disk looks to be : Sandisk Corp WD Black 2018/PC SN720 NVMe SSD
Rand write : fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randwrite -runtime=60 -filename=/dev/nvme1n1p4 test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][w=12.3MiB/s][w=3158 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=177938: Fri Aug 16 17:18:18 2019 write: IOPS=3105, BW=12.1MiB/s (12.7MB/s)(728MiB/60001msec); 0 zone resets slat (nsec): min=1545, max=141856, avg=6784.48, stdev=6188.10 clat (nsec): min=688, max=3539.6k, avg=14665.22, stdev=13601.60 lat (usec): min=9, max=3549, avg=21.66, stdev=16.07 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 19], 90.00th=[ 26], 95.00th=[ 36], | 99.00th=[ 49], 99.50th=[ 52], 99.90th=[ 76], 99.95th=[ 85], | 99.99th=[ 135] bw ( KiB/s): min=10504, max=13232, per=100.00%, avg=12420.01, stdev=439.86, samples=119 iops : min= 2626, max= 3308, avg=3105.00, stdev=109.96, samples=119 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.02%, 4=0.09%, 10=40.60%, 20=43.62%, 50=14.92% lat (usec) : 100=0.73%, 250=0.02%, 500=0.01%, 750=0.01% lat (msec) : 4=0.01% fsync/fdatasync/sync_file_range: sync (nsec): min=12, max=22490, avg=185.20, stdev=255.75 sync percentiles (nsec): | 1.00th=[ 28], 5.00th=[ 35], 10.00th=[ 42], 20.00th=[ 59], | 30.00th=[ 82], 40.00th=[ 109], 50.00th=[ 137], 60.00th=[ 163], | 70.00th=[ 197], 80.00th=[ 253], 90.00th=[ 390], 95.00th=[ 572], | 99.00th=[ 804], 99.50th=[ 820], 99.90th=[ 1144], 99.95th=[ 1208], | 99.99th=[15552] cpu : usr=2.42%, sys=4.91%, ctx=546845, majf=0, minf=12 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=0,186313,0,186313 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): WRITE: bw=12.1MiB/s (12.7MB/s), 12.1MiB/s-12.1MiB/s (12.7MB/s- 12.7MB/s), io=728MiB (763MB), run=60001-60001msec
Disk stats (read/write): nvme1n1: ios=0/375662, merge=0/1492, ticks=0/57478, in_queue=59100, util=97.85%
RBD read
~# fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=read -runtime=60 -pool=kube -rbdname=bench test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=rbd, iodepth=32 fio-3.12 Starting 1 process Jobs: 1 (f=1): [R(1)][69.2%][r=1313MiB/s][r=328 IOPS][eta 00m:08s] test: (groupid=0, jobs=1): err= 0: pid=184386: Fri Aug 16 17:20:45 2019 read: IOPS=144, BW=580MiB/s (608MB/s)(10.0GiB/17668msec) slat (nsec): min=371, max=72009, avg=5416.87, stdev=3649.19 clat (msec): min=6, max=1438, avg=220.83, stdev=192.27 lat (msec): min=6, max=1438, avg=220.83, stdev=192.27 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 23], 10.00th=[ 25], 20.00th=[ 30], | 30.00th=[ 33], 40.00th=[ 47], 50.00th=[ 249], 60.00th=[ 288], | 70.00th=[ 326], 80.00th=[ 372], 90.00th=[ 447], 95.00th=[ 535], | 99.00th=[ 802], 99.50th=[ 844], 99.90th=[ 936], 99.95th=[ 1011], | 99.99th=[ 1435] bw ( KiB/s): min=253952, max=4628480, per=93.74%, avg=556353.83, stdev=851591.15, samples=35 iops : min= 62, max= 1130, avg=135.83, stdev=207.91, samples=35 lat (msec) : 10=0.08%, 20=2.03%, 50=38.40%, 100=2.15%, 250=7.62% lat (msec) : 500=43.48%, 750=4.73%, 1000=1.45% cpu : usr=0.19%, sys=0.24%, ctx=2563, majf=0, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=98.8%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, =64=0.0% issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs): READ: bw=580MiB/s (608MB/s), 580MiB/s-580MiB/s (608MB/s-608MB/s), io=10.0GiB (10.7GB), run=17668-17668msec
Disk stats (read/write): md2: ios=1/1405, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1332/3191, aggrmerge=5911/5187, aggrticks=1031/1257, aggrin_queue=15480, aggrutil=91.95% nvme1n1: ios=0/1183, merge=0/450, ticks=0/141, in_queue=12452, util=69.94% nvme0n1: ios=2665/5199, merge=11822/9925, ticks=2062/2373, in_queue=18508, util=91.95%
RBD write
fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -pool=kube -rbdname=bench test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=rbd, iodepth=32 fio-3.12 Starting 1 process Jobs: 1 (f=1): [W(1)][55.3%][eta 00m:51s] test: (groupid=0, jobs=1): err= 0: pid=181899: Fri Aug 16 17:20:19 2019 write: IOPS=22, BW=90.9MiB/s (95.3MB/s)(5764MiB/63404msec); 0 zone resets slat (usec): min=366, max=4889, avg=802.71, stdev=414.92 clat (msec): min=189, max=7308, avg=1404.62, stdev=1278.48 lat (msec): min=189, max=7308, avg=1405.43, stdev=1278.47 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 376], 10.00th=[ 435], 20.00th=[ 550], | 30.00th=[ 676], 40.00th=[ 776], 50.00th=[ 877], 60.00th=[ 1083], | 70.00th=[ 1351], 80.00th=[ 1921], 90.00th=[ 3473], 95.00th=[ 4597], | 99.00th=[ 5470], 99.50th=[ 6007], 99.90th=[ 6745], 99.95th=[ 7282], | 99.99th=[ 7282] bw ( KiB/s): min= 8192, max=155648, per=100.00%, avg=95454.68, stdev=30757.07, samples=121 iops : min= 2, max= 38, avg=23.27, stdev= 7.52, samples=121 lat (msec) : 250=0.07%, 500=15.82%, 750=22.21%, 1000=18.74% cpu : usr=1.71%, sys=0.15%, ctx=597, majf=0, minf=45137 IO depths : 1=0.1%, 2=0.1%, 4=0.3%, 8=0.6%, 16=1.1%, 32=97.8%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, =64=0.0% issued rwts: total=0,1441,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs): WRITE: bw=90.9MiB/s (95.3MB/s), 90.9MiB/s-90.9MiB/s (95.3MB/s- 95.3MB/s), io=5764MiB (6044MB), run=63404-63404msec
Disk stats (read/write): md2: ios=0/5119, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=30/16383, aggrmerge=20/28021, aggrticks=5/18949, aggrin_queue=63466, aggrutil=90.70% nvme1n1: ios=0/4230, merge=0/1595, ticks=0/1835, in_queue=42440, util=65.48% nvme0n1: ios=60/28536, merge=41/54447, ticks=10/36063, in_queue=84492, util=90.70%
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randread -runtime=60 -filename=/dev/nvme1n1p4 test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=36.3MiB/s][r=9286 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=208060: Fri Aug 16 17:33:01 2019 read: IOPS=9103, BW=35.6MiB/s (37.3MB/s)(2134MiB/60001msec) slat (nsec): min=1384, max=244751, avg=6077.86, stdev=5529.26 clat (usec): min=3, max=8311, avg=101.98, stdev=42.25 lat (usec): min=34, max=8341, avg=108.28, stdev=43.81 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 83], 10.00th=[ 85], 20.00th=[ 90], | 30.00th=[ 92], 40.00th=[ 94], 50.00th=[ 96], 60.00th=[ 100], | 70.00th=[ 104], 80.00th=[ 112], 90.00th=[ 121], 95.00th=[ 141], | 99.00th=[ 182], 99.50th=[ 198], 99.90th=[ 253], 99.95th=[ 297], | 99.99th=[ 2147] bw ( KiB/s): min=33088, max=40928, per=99.96%, avg=36400.00, stdev=1499.29, samples=119 iops : min= 8272, max=10232, avg=9099.98, stdev=374.82, samples=119 lat (usec) : 4=0.01%, 10=0.01%, 50=0.53%, 100=59.74%, 250=39.62% lat (usec) : 500=0.08%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=4.80%, sys=8.33%, ctx=546238, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% issued rwts: total=546228,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs): READ: bw=35.6MiB/s (37.3MB/s), 35.6MiB/s-35.6MiB/s (37.3MB/s- 37.3MB/s), io=2134MiB (2237MB), run=60001-60001msec
Disk stats (read/write): nvme1n1: ios=545111/3564, merge=0/1430, ticks=54388/943, in_queue=60924, util=100.00%
Le vendredi 16 août 2019 à 18:06 +0300, vitalif@yourcmc.ru a écrit :
Now to go for "apples to apples" either run
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randwrite -runtime=60 -filename=/dev/nvmeXXXXXXXXX
to compare with the single-threaded RBD random write result (the test is destructive, so use a separate partition without data)
...Or run
fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=write -runtime=60 -pool=kube -rbdname=bench
to compare with your dd's linear write result.
58 single-threaded random iops for NVMes is pretty sad either way. Are you NVMe's server ones? Do they have capacitors? :) in the response to a likely question about what they are I'll just post my link here again :) https://yourcmc.ru/wiki/Ceph_performance
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
- libaio randwrite - libaio randread
- libaio randwrite on mapped rbd - libaio randread on mapped rbd
- rbd read - rbd write
recheck RBD with RAND READ / RAND WRITE you're again comparing RANDOM and NON-RANDOM I/O your SSDs aren't that bad, 3000 single-thread iops isn't the worst possible performance
fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=randwrite -runtime=60 -pool=kube -rbdname=bench WRITE: bw=89.6MiB/s (93.9MB/s), 89.6MiB/s-89.6MiB/s (93.9MB/s- 93.9MB/s), io=5548MiB (5817MB), run=61935-61935msec fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=randread -runtime=60 -pool=kube -rbdname=bench READ: bw=139MiB/s (146MB/s), 139MiB/s-139MiB/s (146MB/s-146MB/s), io=8404MiB (8812MB), run=60443-60443msec this is great. I don't get why I can't get the same in the mapped rbd. Le vendredi 16 août 2019 à 21:04 +0300, vitalif@yourcmc.ru a écrit :
- libaio randwrite - libaio randread - libaio randwrite on mapped rbd - libaio randread on mapped rbd - rbd read - rbd write
recheck RBD with RAND READ / RAND WRITE
you're again comparing RANDOM and NON-RANDOM I/O
your SSDs aren't that bad, 3000 single-thread iops isn't the worst possible performance _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
And once more you're checking random I/O with 4 MB !!! block size. Now recheck it with bs=4k.
RBD fio -ioengine=rbd -name=test -bs=4k -iodepth=32 -rw=randread -runtime=60 -pool=kube -rbdname=bench READ: bw=21.8MiB/s (22.8MB/s), 21.8MiB/s-21.8MiB/s (22.8MB/s-22.8MB/s), io=1308MiB (1371MB), run=60011-60011msec fio -ioengine=rbd -name=test -bs=4k -iodepth=32 -rw=randwrite -runtime=60 -pool=kube -rbdname=bench WRITE: bw=5968KiB/s (6111kB/s), 5968KiB/s-5968KiB/s (6111kB/s- 6111kB/s), io=350MiB (367MB), run=60022-60022msec mapped rbd : fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randread -runtime=60 -filename=/dev/rbd/kube/bench READ: bw=785KiB/s (804kB/s), 785KiB/s-785KiB/s (804kB/s-804kB/s), io=46.0MiB (48.3MB), run=60008-60008msec old hardware raw disk: fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randwrite -runtime=60 -filename=/dev/sda4 WRITE: bw=107MiB/s (112MB/s), 107MiB/s-107MiB/s (112MB/s-112MB/s), io=6401MiB (6712MB), run=60001-60001msec fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randread -runtime=60 -filename=/dev/sda4 READ: bw=36.8MiB/s (38.6MB/s), 36.8MiB/s-36.8MiB/s (38.6MB/s-38.6MB/s), io=2208MiB (2315MB), run=60001-60001msec new hardware raw disk : fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randwrite -runtime=60 -filename=/dev/nvme1n1p4 WRITE: bw=12.1MiB/s (12.7MB/s), 12.1MiB/s-12.1MiB/s (12.7MB/s- 12.7MB/s), io=728MiB (763MB), run=60001-60001msec fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randread -runtime=60 -filename=/dev/nvme1n1p4 READ: bw=35.6MiB/s (37.3MB/s), 35.6MiB/s-35.6MiB/s (37.3MB/s-37.3MB/s), io=2134MiB (2237MB), run=60001-60001msec Le vendredi 16 août 2019 à 21:34 +0300, vitalif@yourcmc.ru a écrit :
And once more you're checking random I/O with 4 MB !!! block size.
Now recheck it with bs=4k.
on a new ceph cluster with the same software and config (ansible) on the old hardware. 2 replica, 1 host, 4 osd. RBD fio -ioengine=rbd -name=test -bs=4k -iodepth=32 -rw=randread -runtime=60 -pool=kube -rbdname=bench READ: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s), io=7189MiB (7538MB), run=60001-60001msec fio -ioengine=rbd -name=test -bs=4k -iodepth=32 -rw=randwrite -runtime=60 -pool=kube -rbdname=bench WRITE: bw=42.0MiB/s (44.1MB/s), 42.0MiB/s-42.0MiB/s (44.1MB/s- 44.1MB/s), io=2522MiB (2645MB), run=60004-60004msec MAPPED RBD fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randread -direct=1 -runtime=60 -filename=/dev/rbd/kube/bench READ: bw=184MiB/s (193MB/s), 184MiB/s-184MiB/s (193MB/s-193MB/s), io=10.0GiB (10.7GB), run=55613-55613msec fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randwrite -direct=1 -runtime=60 -filename=/dev/rbd/kube/bench WRITE: bw=46.9MiB/s (49.2MB/s), 46.9MiB/s-46.9MiB/s (49.2MB/s- 49.2MB/s), io=2814MiB (2950MB), run=60002-60002msec looks much better old the old one on the cluster with the new hardware. 2 replica 3 host 6 osd. fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randread -direct=1 -runtime=60 -filename=/dev/rbd/kube/bench READ: bw=31.1MiB/s (32.6MB/s), 31.1MiB/s-31.1MiB/s (32.6MB/s-32.6MB/s), io=1866MiB (1957MB), run=60010-60010msec fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randwrite -direct=1 -runtime=60 -filename=/dev/rbd/kube/bench WRITE: bw=10.5MiB/s (11.0MB/s), 10.5MiB/s-10.5MiB/s (11.0MB/s- 11.0MB/s), io=631MiB (662MB), run=60021-60021msec => New hardware : 32.6MB/s READ / 10.5MiB WRITE => Old hardware : 184MiB/s READ / 46.9MiB WRITE No discussion ? I suppose I will keep the old hardware. What do you think ? :D Le vendredi 16 août 2019 à 21:17 +0200, Olivier AUDRY a écrit :
RBD fio -ioengine=rbd -name=test -bs=4k -iodepth=32 -rw=randread -runtime=60 -pool=kube -rbdname=bench READ: bw=21.8MiB/s (22.8MB/s), 21.8MiB/s-21.8MiB/s (22.8MB/s- 22.8MB/s), io=1308MiB (1371MB), run=60011-60011msec
fio -ioengine=rbd -name=test -bs=4k -iodepth=32 -rw=randwrite -runtime=60 -pool=kube -rbdname=bench WRITE: bw=5968KiB/s (6111kB/s), 5968KiB/s-5968KiB/s (6111kB/s- 6111kB/s), io=350MiB (367MB), run=60022-60022msec
mapped rbd : fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randread -runtime=60 -filename=/dev/rbd/kube/bench READ: bw=785KiB/s (804kB/s), 785KiB/s-785KiB/s (804kB/s-804kB/s), io=46.0MiB (48.3MB), run=60008-60008msec
old hardware raw disk:
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randwrite -runtime=60 -filename=/dev/sda4 WRITE: bw=107MiB/s (112MB/s), 107MiB/s-107MiB/s (112MB/s-112MB/s), io=6401MiB (6712MB), run=60001-60001msec
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randread -runtime=60 -filename=/dev/sda4 READ: bw=36.8MiB/s (38.6MB/s), 36.8MiB/s-36.8MiB/s (38.6MB/s- 38.6MB/s), io=2208MiB (2315MB), run=60001-60001msec
new hardware raw disk :
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randwrite -runtime=60 -filename=/dev/nvme1n1p4 WRITE: bw=12.1MiB/s (12.7MB/s), 12.1MiB/s-12.1MiB/s (12.7MB/s- 12.7MB/s), io=728MiB (763MB), run=60001-60001msec
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1 -rw=randread -runtime=60 -filename=/dev/nvme1n1p4 READ: bw=35.6MiB/s (37.3MB/s), 35.6MiB/s-35.6MiB/s (37.3MB/s- 37.3MB/s), io=2134MiB (2237MB), run=60001-60001msec
Le vendredi 16 août 2019 à 21:34 +0300, vitalif@yourcmc.ru a écrit :
And once more you're checking random I/O with 4 MB !!! block size.
Now recheck it with bs=4k.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
on a new ceph cluster with the same software and config (ansible) on the old hardware. 2 replica, 1 host, 4 osd.
=> New hardware : 32.6MB/s READ / 10.5MiB WRITE => Old hardware : 184MiB/s READ / 46.9MiB WRITE
No discussion ? I suppose I will keep the old hardware. What do you think ? :D
In fact I don't really believe in 184 MB/s random reads with Ceph with 4 OSDs, it's a very cool result if it's true. Does the "new cluster on the old hardware" consist of only 1 host? Did you test reads before you actually wrote anything into the image so it was empty and reads were fast because of that?
Write and read with 2 hosts 4 osd : mkfs.ext4 /dev/rbd/kube/bench mount /dev/rbd/kube/bench /mnt/ dd if=/dev/zero of=test bs=8192k count=1000 oflag=direct 8388608000 bytes (8.4 GB, 7.8 GiB) copied, 117.541 s, 71.4 MB/s fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randwrite -direct=1 -runtime=60 -filename=/dev/rbd/kube/bench WRITE: bw=45.3MiB/s (47.5MB/s), 45.3MiB/s-45.3MiB/s (47.5MB/s- 47.5MB/s), io=2718MiB (2850MB), run=60003-60003msec fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randread -direct=1 -runtime=60 -filename=/dev/rbd/kube/bench READ: bw=187MiB/s (197MB/s), 187MiB/s-187MiB/s (197MB/s-197MB/s), io=10.0GiB (10.7GB), run=54636-54636msec pgbench before : 10 transaction per second pgbench after : 355 transaction per second So yes it's better. SSD are INTEL SSDSC2BB48 0370. Le samedi 17 août 2019 à 01:55 +0300, vitalif@yourcmc.ru a écrit :
on a new ceph cluster with the same software and config (ansible) on the old hardware. 2 replica, 1 host, 4 osd.
=> New hardware : 32.6MB/s READ / 10.5MiB WRITE => Old hardware : 184MiB/s READ / 46.9MiB WRITE
No discussion ? I suppose I will keep the old hardware. What do you think ? :D
In fact I don't really believe in 184 MB/s random reads with Ceph with 4 OSDs, it's a very cool result if it's true.
Does the "new cluster on the old hardware" consist of only 1 host? Did you test reads before you actually wrote anything into the image so it was empty and reads were fast because of that? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This probably muddies the water. Note Active cluster with around 22 read/write IOPS and 200kB read/write A CephFS mounted with 3 hosts 6 osd per host with 8G public and 10G private networking for Ceph. No SSDs and mostly WD Red 1T 2.5" drives some are HGST 1T 7200. root@blade7:~# fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randwrite -direct=1 -runtime=60 -filename=/mnt/pve/cephfs/test test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 fio-3.12 Starting 1 process test: you need to specify size= fio: pid=0, err=22/file:filesetup.c:952, func=total_file_size, error=Invalid argument Run status group 0 (all jobs): root@blade7:~# fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randwrite -direct=1 -runtime=60 -size=10G -filename=/mnt/pve/cephfs/test test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 fio-3.12 Starting 1 process test: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=0): [f(1)][100.0%][w=580KiB/s][w=145 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=3561674: Sat Aug 17 09:20:22 2019 write: IOPS=2262, BW=9051KiB/s (9268kB/s)(538MiB/60845msec); 0 zone resets slat (usec): min=8, max=35648, avg=40.01, stdev=97.51 clat (usec): min=954, max=2854.3k, avg=14090.15, stdev=100194.83 lat (usec): min=994, max=2854.3k, avg=14130.65, stdev=100195.40 clat percentiles (usec): | 1.00th=[ 1254], 5.00th=[ 1450], 10.00th=[ 1582], | 20.00th=[ 1795], 30.00th=[ 2008], 40.00th=[ 2245], | 50.00th=[ 2540], 60.00th=[ 2933], 70.00th=[ 3392], | 80.00th=[ 4228], 90.00th=[ 7767], 95.00th=[ 35914], | 99.00th=[ 254804], 99.50th=[ 616563], 99.90th=[1652556], | 99.95th=[2122318], 99.99th=[2600469] bw ( KiB/s): min= 48, max=44408, per=100.00%, avg=10387.54, stdev=10384.94, samples=106 iops : min= 12, max=11102, avg=2596.88, stdev=2596.23, samples=106 lat (usec) : 1000=0.01% lat (msec) : 2=29.82%, 4=47.95%, 10=14.23%, 20=2.43%, 50=1.34% lat (msec) : 100=2.68%, 250=0.53%, 500=0.40%, 750=0.20%, 1000=0.14% cpu : usr=1.45%, sys=6.36%, ctx=151946, majf=0, minf=280 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, =64=0.0% issued rwts: total=0,137674,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs): WRITE: bw=9051KiB/s (9268kB/s), 9051KiB/s-9051KiB/s (9268kB/s-9268kB/s), io=538MiB (564MB), run=60845-60845msec This is on the same system with a RBD mapped file system root@blade7:/mnt# fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randwrite -direct=1 -runtime=60 -size=10G -filename=/mnt/image0/test test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 fio-3.12 Starting 1 process test: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][4.5%][w=4KiB/s][w=1 IOPS][eta 21m:30s] test: (groupid=0, jobs=1): err= 0: pid=3567399: Sat Aug 17 09:38:55 2019 write: IOPS=1935, BW=7744KiB/s (7930kB/s)(462MiB/61143msec); 0 zone resets slat (usec): min=9, max=700161, avg=65.17, stdev=2092.54 clat (usec): min=954, max=2578.6k, avg=16457.67, stdev=109995.03 lat (usec): min=1021, max=2578.6k, avg=16523.42, stdev=110014.91 clat percentiles (usec): | 1.00th=[ 1254], 5.00th=[ 1434], 10.00th=[ 1549], | 20.00th=[ 1745], 30.00th=[ 1909], 40.00th=[ 2114], | 50.00th=[ 2376], 60.00th=[ 2704], 70.00th=[ 3228], | 80.00th=[ 4080], 90.00th=[ 8717], 95.00th=[ 53216], | 99.00th=[ 291505], 99.50th=[ 675283], 99.90th=[1669333], | 99.95th=[2231370], 99.99th=[2365588] bw ( KiB/s): min= 8, max=35968, per=100.00%, avg=9015.64, stdev=8402.84, samples=105 iops : min= 2, max= 8992, avg=2253.90, stdev=2100.72, samples=105 lat (usec) : 1000=0.01% lat (msec) : 2=34.85%, 4=44.49%, 10=11.54%, 20=1.84%, 50=1.81% lat (msec) : 100=3.27%, 250=1.13%, 500=0.42%, 750=0.19%, 1000=0.08% cpu : usr=1.42%, sys=6.63%, ctx=123309, majf=0, minf=283 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, =64=0.0% issued rwts: total=0,118371,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs): WRITE: bw=7744KiB/s (7930kB/s), 7744KiB/s-7744KiB/s (7930kB/s-7930kB/s), io=462MiB (485MB), run=61143-61143msec Disk stats (read/write): rbd0: ios=0/118670, merge=0/9674, ticks=0/1894238, in_queue=1651008, util=33.33% On 17/8/19 8:46 am, Olivier AUDRY wrote:
Write and read with 2 hosts 4 osd :
mkfs.ext4 /dev/rbd/kube/bench mount /dev/rbd/kube/bench /mnt/ dd if=/dev/zero of=test bs=8192k count=1000 oflag=direct 8388608000 bytes (8.4 GB, 7.8 GiB) copied, 117.541 s, 71.4 MB/s
fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randwrite -direct=1 -runtime=60 -filename=/dev/rbd/kube/bench WRITE: bw=45.3MiB/s (47.5MB/s), 45.3MiB/s-45.3MiB/s (47.5MB/s- 47.5MB/s), io=2718MiB (2850MB), run=60003-60003msec
fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randread -direct=1 -runtime=60 -filename=/dev/rbd/kube/bench READ: bw=187MiB/s (197MB/s), 187MiB/s-187MiB/s (197MB/s-197MB/s), io=10.0GiB (10.7GB), run=54636-54636msec
pgbench before : 10 transaction per second pgbench after : 355 transaction per second
So yes it's better. SSD are INTEL SSDSC2BB48 0370.
Le samedi 17 août 2019 à 01:55 +0300, vitalif@yourcmc.ru a écrit :
on a new ceph cluster with the same software and config (ansible) on the old hardware. 2 replica, 1 host, 4 osd.
=> New hardware : 32.6MB/s READ / 10.5MiB WRITE => Old hardware : 184MiB/s READ / 46.9MiB WRITE
No discussion ? I suppose I will keep the old hardware. What do you think ? :D In fact I don't really believe in 184 MB/s random reads with Ceph with 4 OSDs, it's a very cool result if it's true.
Does the "new cluster on the old hardware" consist of only 1 host? Did you test reads before you actually wrote anything into the image so it was empty and reads were fast because of that? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Ilya Dryomov
-
Mike O'Connor
-
Olivier AUDRY
-
Paul Emmerich
-
vitalif＠yourcmc.ru
-
Vitaliy Filippov