14 Aug
2019
14 Aug
'19
5:07 p.m.
hello
I got a huge difference of read/write performance on my ceph cluster
when I use rbd.
rbd bench reach the limit of my cluster (1gbps network) when the
performance in mapped rbd is very low. 30MB/s.
ceph version : ceph version 14.2.2
(4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)
kernel version : 4.19.0-5-amd64
ceph -s
cluster:
id: fdf6eba0-cde9-401c-8901-751eb1e8b7d2
health: HEALTH_OK
services:
mon: 3 daemons, quorum gra3-k8s001,sbg2-k8s001,rbx4-k8s001 (age
41h)
mgr: sbg2-k8s001(active, since 46h), standbys: gra3-k8s001, rbx4-
k8s001
osd: 10 osds: 10 up (since 41h), 10 in (since 41h)
rgw: 3 daemons active (gra3-k8s001, rbx4-k8s001, sbg2-k8s001)
data:
pools: 8 pools, 240 pgs
objects: 109.14k objects, 229 GiB
usage: 694 GiB used, 3.0 TiB / 3.7 TiB avail
pgs: 240 active+clean
here the bench
rbd create --size=10G kube/bench
rbd feature disable kube/bench object-map fast-diff deep-flatten
rbd -p kube bench kube/bench --io-type write --io-size 8192 --io-
threads 256 --io-total 10G --io-pattern seq
elapsed: 110 ops: 1310720 ops/sec: 11884.39 bytes/sec:
97356926.64
this reach the 1gbps limit of my network. Looks good.
let's test random write
rbd -p kube bench kube/bench --io-type write --io-size 8192 --io-
threads 256 --io-total 10G --io-pattern rand
elapsed: 125 ops: 1310720 ops/sec: 10416.31 bytes/sec:
85330446.58
this reach the 1gbps limit of my network. Looks good.
rbd -p kube bench kube/bench --io-type read --io-size 8192 --io-threads
256 --io-total 10G --io-pattern seq
elapsed: 40 ops: 1310720 ops/sec: 32712.03 bytes/sec:
267976974.35
this is great perf !
let's try rand
rbd -p kube bench kube/bench --io-type read --io-size 8192 --io-threads
256 --io-total 10G --io-pattern rand
elapsed: 29 ops: 1310720 ops/sec: 43859.78 bytes/sec:
359299337.92
this is great perf !
But when I mount the image :
rbd map bench --pool kube --name client.admin
/sbin/mkfs.ext4 /dev/rbd/kube/bench
mount /dev/rbd/kube/bench /mnt/
cd /mnt/
dd if=/dev/zero of=test bs=8192k count=100 oflag=direct
838860800 bytes (839 MB, 800 MiB) copied, 24.6185 s, 34.1 MB/s
34.1MB/s vs 85MB/s ....
14 Aug
14 Aug
5:18 p.m.
On Wed, Aug 14, 2019 at 2:38 PM Olivier AUDRY <olivier(a)nmlq.fr> wrote:
let's test random write
rbd -p kube bench kube/bench --io-type write --io-size 8192 --io-threads 256 --io-total
10G --io-pattern rand
elapsed: 125 ops: 1310720 ops/sec: 10416.31 bytes/sec: 85330446.58
dd if=/dev/zero of=test bs=8192k count=100 oflag=direct
838860800 bytes (839 MB, 800 MiB) copied, 24.6185 s, 34.1 MB/s
34.1MB/s vs 85MB/s ....
34 apples vs. 85 oranges
You are comparing 256 threads with a huge queue depth vs a single
thread with a normal queue depth.
Use fio on the mounted rbd to get better control over what it's doing
Paul
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:26 p.m.
On Wed, Aug 14, 2019 at 2:49 PM Paul Emmerich <paul.emmerich(a)croit.io> wrote:
On Wed, Aug 14, 2019 at 2:38 PM Olivier AUDRY <olivier(a)nmlq.fr> wrote:
When you said mounted, did you mean mapped or "a filesystem mounted on
top of a mapped rbd"?
There is no filesystem in "rbd bench" tests, so fio should be used on
a raw block device. It still won't be completely apples to apples
because in "rbd bench" or fio's rbd engine (--ioengine=rbd) case there
is no block layer either, but it is closer...
Thanks,
Ilya
let's test random write
rbd -p kube bench kube/bench --io-type write --io-size 8192 --io-threads 256 --io-total
10G --io-pattern rand
elapsed: 125 ops: 1310720 ops/sec: 10416.31 bytes/sec: 85330446.58
dd if=/dev/zero of=test bs=8192k count=100 oflag=direct
838860800 bytes (839 MB, 800 MiB) copied, 24.6185 s, 34.1 MB/s
34.1MB/s vs 85MB/s ....
34 apples vs. 85 oranges
You are comparing 256 threads with a huge queue depth vs a single
thread with a normal queue depth.
Use fio on the mounted rbd to get better control over what it's doing
6:40 p.m.
hello
I mean a filesystem mounted on top of a mapped rbd
rbd create --size=10G kube/benchrbd feature disable kube/bench object-
map fast-diff deep-flattenrbd map bench --pool kube --name
client.admin/sbin/mkfs.ext4 /dev/rbd/kube/bench mount
/dev/rbd/kube/bench /mnt/cd /mnt/
about the bench I did. I try to get apples in both side. (I hope) :
block size : 4kthread : 1size of data : 1G
Writes are great.
rbd -p kube bench kube/bench --io-type write --io-threads 1 --io-total
1G --io-pattern seq
elapsed: 12 ops: 262144 ops/sec: 20758.70 bytes/sec:
85027625.70
rbd -p kube bench kube/bench --io-type write --io-threads 1 --io-total
10G --io-pattern rand
elapsed: 14 ops: 262144 ops/sec: 17818.16 bytes/sec:
72983201.32
Reads are very very slow :rbd -p kube bench kube/bench --io-type read
--io-threads 1 --io-total 1G --io-pattern
randelapsed: 445 ops: 81216 ops/sec: 182.37 bytes/sec:
747006.15
rbd -p kube bench kube/bench --io-type read --io-threads 1 --io-total
1G --io-pattern
seqelapsed: 14 ops: 14153 ops/sec: 957.57 bytes/sec:
3922192.15
Perhaps I'm hitting this 'issue' :
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028878.html
For the record :
I got an old cluster with vm in ceph 10.2.11. With a dd bench I reach
the cluster limitation.
dd if=/dev/zero of=test bs=4M count=250 oflag=direct1048576000 bytes
(1.0 GB) copied, 11.5469 s, 90.8 MB/s
and pgbench gives me 200 transaction per second
On the new cluster with containers running on fs on top of a mapped rbd
and ceph nautilus I got :dd if=/dev/zero of=test bs=4M count=250
oflag=direct1048576000 bytes (1.0 GB, 1000 MiB) copied, 27.0351 s, 38.8
MB/s
and pgbench gives me 10 transactions per second.
something it not ok somewhere :)
oau
Le mercredi 14 août 2019 à 15:56 +0200, Ilya Dryomov a écrit :
On Wed, Aug 14, 2019 at 2:49 PM Paul Emmerich
<paul.emmerich(a)croit.io
wrote:
On Wed, Aug 14, 2019 at 2:38 PM Olivier AUDRY <olivier(a)nmlq.fr>
wrote:
When you said mounted, did you mean mapped or "a filesystem mounted
on
top of a mapped rbd"?
There is no filesystem in "rbd bench" tests, so fio should be used on
a raw block device. It still won't be completely apples to apples
because in "rbd bench" or fio's rbd engine (--ioengine=rbd) case
there
is no block layer either, but it is closer...
Thanks,
Ilya let's test random write
rbd -p kube bench kube/bench --io-type write --io-size 8192 --io-
threads 256 --io-total 10G --io-pattern rand
elapsed: 125 ops: 1310720 ops/sec: 10416.31 bytes/sec:
85330446.58
dd if=/dev/zero of=test bs=8192k count=100 oflag=direct
838860800 bytes (839 MB, 800 MiB) copied, 24.6185 s, 34.1 MB/s
34.1MB/s vs 85MB/s ....
34 apples vs. 85 oranges
You are comparing 256 threads with a huge queue depth vs a single
thread with a normal queue depth.
Use fio on the mounted rbd to get better control over what it's
doing 
16 Aug
16 Aug
2:46 a.m.
rbd -p kube bench kube/bench --io-type write
--io-threads 1 --io-total
10G --io-pattern rand
elapsed: 14 ops: 262144 ops/sec: 17818.16 bytes/sec: 72983201.32
It's a totally unreal number. Something is wrong with the test.
Test it with `fio` please:
fio -ioengine=rbd -name=test -bs=4k -iodepth=1 -rw=randwrite -runtime=60
-pool=kube -rbdname=bench
Reads are very very slow:
elapsed: 445 ops: 81216 ops/sec: 182.37 bytes/sec: 747006.15
elapsed: 14 ops: 14153 ops/sec: 957.57 bytes/sec: 3922192.15
This is closer to the reality.
--
With best regards,
Vitaliy Filippov
4:01 p.m.
hello
here the result :
fio --ioengine=rbd --name=test --bs=4k --iodepth=1 --rw=randwrite --
runtime=60 -pool=kube -rbdname=bench
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=rbd, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=248KiB/s][w=62 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1903256: Fri Aug 16 13:22:59
2019
write: IOPS=58, BW=232KiB/s (238kB/s)(13.6MiB/60011msec); 0 zone
resets
slat (usec): min=9, max=351, avg=52.82, stdev=23.02
clat (usec): min=1264, max=96970, avg=17156.70, stdev=6713.88
lat (usec): min=1276, max=97050, avg=17209.52, stdev=6715.06
clat percentiles (usec):
| 1.00th=[ 2933], 5.00th=[ 3884], 10.00th=[11863],
20.00th=[13304],
| 30.00th=[13960], 40.00th=[14484], 50.00th=[15008],
60.00th=[20579],
| 70.00th=[22152], 80.00th=[23987], 90.00th=[25297],
95.00th=[25822],
| 99.00th=[26346], 99.50th=[27395], 99.90th=[71828],
99.95th=[82314],
| 99.99th=[96994]
bw ( KiB/s): min= 104, max= 272, per=100.00%, avg=232.17,
stdev=19.55, samples=120
iops : min= 26, max= 68, avg=57.97, stdev= 4.88,
samples=120
lat (msec) : 2=0.06%, 4=5.51%, 10=3.41%, 20=50.22%, 50=40.69%
lat (msec) : 100=0.11%
cpu : usr=0.44%, sys=0.27%, ctx=3489, majf=0, minf=3582
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
issued rwts: total=0,3485,0,0
short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=232KiB/s (238kB/s), 232KiB/s-232KiB/s (238kB/s-238kB/s),
io=13.6MiB (14.3MB), run=60011-60011msec
Disk stats (read/write):
md2: ios=3/4611, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=3/8532, aggrmerge=0/2364, aggrticks=0/1346, aggrin_queue=52504,
aggrutil=88.03%
nvme1n1: ios=3/8529, merge=0/2295, ticks=1/1347, in_queue=52932,
util=88.03%
nvme0n1: ios=3/8535, merge=1/2434, ticks=0/1346, in_queue=52076,
util=86.52%
for you information my disk setup is :
2 500G nvme disk with :
10G raid 1 partition with the OS
160G raid0 partition for local docker data
387.00g partition on each disk for the osd.
I got 5 physical devices with 12 core cpu and 32go RAM. Ceph is running
on dedicated 1Gbps network.
I got quite the same hardware setup with ceph 10.2.11 and I got much
better performance.
oau
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
rbd0 252:0 0 1G 0
disk /var/lib/kubelet/pods/db24eb3b-650c-42f3-bdf0-
92ea0eaf37d8/volumes/kubernetes.io~csi/pvc-e70f3d74-c7bd-4652-983b-
de3874d36117/mou
rbd1 252:16 0 1G 0
disk /var/lib/kubelet/pods/a1b0b156-0bcc-4d0c-b2f8-
26a74337baed/volumes/kubernetes.io~csi/pvc-14460f8a-da5a-44e3-a033-
a3bf5054f967/mou
rbd2 252:32 0 8G 0
disk /var/lib/kubelet/pods/0a5e3745-c6c2-49a9-971a-
3ddac59af66c/volumes/kubernetes.io~csi/pvc-b8907f44-58d6-4599-b189-
fafe65daed09/mou
rbd3 252:48 0 50G 0
disk /var/lib/kubelet/pods/84df04af-a7dd-4035-a9a2-
22d3d315fa60/volumes/kubernetes.io~csi/pvc-faf5cefa-ecce-450a-9f5c-
42e7ca7d7fc2/mou
rbd4 252:64 0 1G 0
disk /var/lib/kubelet/pods/872a3d7d-63d5-4567-86f6-
bedab3fe0ad3/volumes/kubernetes.io~csi/pvc-83ea4013-b936-469b-b20f-
7e703db2c871/mou
nvme0n1 259:0 0 477G 0 disk
├─nvme0n1p1 259:1 0 511M 0 part /boot/efi
├─nvme0n1p2 259:2 0 9.8G 0 part
│ └─md2 9:2 0 9.8G 0 raid1 /
├─nvme0n1p3 259:3 0 79.5G 0 part
│ └─md3 9:3 0 158.9G 0 raid0
│ └─datavg-dockerlv 253:0 0 30G 0 lvm /var/lib/docker
└─nvme0n1p4 259:8 0 387.2G 0 part
└─ceph--52ce0eb9--9e69--4f29--8b87--9ab3fbb5df3e-osd--block
--c26bdb06--2325--4fcd--9b8c--77a93ab46de0
253:1 0 387G 0 lvm
nvme1n1 259:4 0 477G 0 disk
├─nvme1n1p1 259:5 0 511M 0 part
├─nvme1n1p2 259:6 0 9.8G 0 part
│ └─md2 9:2 0 9.8G 0 raid1 /
├─nvme1n1p3 259:7 0 79.5G 0 part
│ └─md3 9:3 0 158.9G 0 raid0
│ └─datavg-dockerlv 253:0 0 30G 0 lvm /var/lib/docker
└─nvme1n1p4 259:9 0 387.2G 0 part
└─ceph--4ef4eb77--4b23--4ffd--b7d2--0fd2cfa5e568-osd--block
--28435fba--91c7--4424--99f2--ea86709a87ca
253:2 0 387G 0 lvm
Le vendredi 16 août 2019 à 01:16 +0300, Vitaliy Filippov a écrit :
rbd -p kube
bench kube/bench --io-type write --io-threads 1 --io-
total
10G --io-pattern rand
elapsed: 14 ops: 262144 ops/sec: 17818.16 bytes/sec:
72983201.32
It's a totally unreal number. Something is wrong with the test.
Test it with `fio` please:
fio -ioengine=rbd -name=test -bs=4k -iodepth=1 -rw=randwrite
-runtime=60
-pool=kube -rbdname=bench
Reads are very very slow:
elapsed: 445 ops: 81216 ops/sec: 182.37 bytes/sec:
747006.15
elapsed: 14 ops: 14153 ops/sec: 957.57 bytes/sec:
3922192.15
This is closer to the reality.
6:52 p.m.
hello
just for the record the nvme disk are pretty fast.
dd if=/dev/zero of=test bs=8192k count=100 oflag=direct
100+0 records in
100+0 records out
838860800 bytes (839 MB, 800 MiB) copied, 0.49474 s, 1.7 GB/s
oau
Le vendredi 16 août 2019 à 13:31 +0200, Olivier AUDRY a écrit :
hello
here the result :
fio --ioengine=rbd --name=test --bs=4k --iodepth=1 --rw=randwrite --
runtime=60 -pool=kube -rbdname=bench
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=rbd, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=248KiB/s][w=62 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1903256: Fri Aug 16 13:22:59
2019
write: IOPS=58, BW=232KiB/s (238kB/s)(13.6MiB/60011msec); 0 zone
resets
slat (usec): min=9, max=351, avg=52.82, stdev=23.02
clat (usec): min=1264, max=96970, avg=17156.70, stdev=6713.88
lat (usec): min=1276, max=97050, avg=17209.52, stdev=6715.06
clat percentiles (usec):
| 1.00th=[ 2933], 5.00th=[ 3884], 10.00th=[11863],
20.00th=[13304],
| 30.00th=[13960], 40.00th=[14484], 50.00th=[15008],
60.00th=[20579],
| 70.00th=[22152], 80.00th=[23987], 90.00th=[25297],
95.00th=[25822],
| 99.00th=[26346], 99.50th=[27395], 99.90th=[71828],
99.95th=[82314],
| 99.99th=[96994]
bw ( KiB/s): min= 104, max= 272, per=100.00%, avg=232.17,
stdev=19.55, samples=120
iops : min= 26, max= 68, avg=57.97, stdev= 4.88,
samples=120
lat (msec) : 2=0.06%, 4=5.51%, 10=3.41%, 20=50.22%, 50=40.69%
lat (msec) : 100=0.11%
cpu : usr=0.44%, sys=0.27%, ctx=3489, majf=0, minf=3582
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
=64=0.0%
submit : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
issued rwts: total=0,3485,0,0
short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=232KiB/s (238kB/s), 232KiB/s-232KiB/s (238kB/s-238kB/s),
io=13.6MiB (14.3MB), run=60011-60011msec
Disk stats (read/write):
md2: ios=3/4611, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=3/8532, aggrmerge=0/2364, aggrticks=0/1346,
aggrin_queue=52504, aggrutil=88.03%
nvme1n1: ios=3/8529, merge=0/2295, ticks=1/1347, in_queue=52932,
util=88.03%
nvme0n1: ios=3/8535, merge=1/2434, ticks=0/1346, in_queue=52076,
util=86.52%
for you information my disk setup is :
2 500G nvme disk with :
10G raid 1 partition with the OS
160G raid0 partition for local docker data
387.00g partition on each disk for the osd.
I got 5 physical devices with 12 core cpu and 32go RAM. Ceph is
running on dedicated 1Gbps network.
I got quite the same hardware setup with ceph 10.2.11 and I got much
better performance.
oau
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
rbd0 252:0 0 1G 0
disk /var/lib/kubelet/pods/db24eb3b-650c-42f3-bdf0-
92ea0eaf37d8/volumes/kubernetes.io~csi/pvc-e70f3d74-c7bd-4652-983b-
de3874d36117/mou
rbd1 252:16 0 1G 0
disk /var/lib/kubelet/pods/a1b0b156-0bcc-4d0c-b2f8-
26a74337baed/volumes/kubernetes.io~csi/pvc-14460f8a-da5a-44e3-a033-
a3bf5054f967/mou
rbd2 252:32 0 8G 0
disk /var/lib/kubelet/pods/0a5e3745-c6c2-49a9-971a-
3ddac59af66c/volumes/kubernetes.io~csi/pvc-b8907f44-58d6-4599-b189-
fafe65daed09/mou
rbd3 252:48 0 50G 0
disk /var/lib/kubelet/pods/84df04af-a7dd-4035-a9a2-
22d3d315fa60/volumes/kubernetes.io~csi/pvc-faf5cefa-ecce-450a-9f5c-
42e7ca7d7fc2/mou
rbd4 252:64 0 1G 0
disk /var/lib/kubelet/pods/872a3d7d-63d5-4567-86f6-
bedab3fe0ad3/volumes/kubernetes.io~csi/pvc-83ea4013-b936-469b-b20f-
7e703db2c871/mou
nvme0n1 259:0 0 477G 0 disk
├─nvme0n1p1 259:1 0 511M 0 part /boot/efi
├─nvme0n1p2 259:2 0 9.8G 0 part
│ └─md2 9:2 0 9.8G 0 raid1 /
├─nvme0n1p3 259:3 0 79.5G 0 part
│ └─md3 9:3 0 158.9G 0 raid0
│ └─datavg-dockerlv 253:0 0 30G 0 lvm /var/lib/docker
└─nvme0n1p4 259:8 0 387.2G 0 part
└─ceph--52ce0eb9--9e69--4f29--8b87--9ab3fbb5df3e-osd--block
--c26bdb06--2325--4fcd--9b8c--77a93ab46de0
253:1 0 387G 0 lvm
nvme1n1 259:4 0 477G 0 disk
├─nvme1n1p1 259:5 0 511M 0 part
├─nvme1n1p2 259:6 0 9.8G 0 part
│ └─md2 9:2 0 9.8G 0 raid1 /
├─nvme1n1p3 259:7 0 79.5G 0 part
│ └─md3 9:3 0 158.9G 0 raid0
│ └─datavg-dockerlv 253:0 0 30G 0 lvm /var/lib/docker
└─nvme1n1p4 259:9 0 387.2G 0 part
└─ceph--4ef4eb77--4b23--4ffd--b7d2--0fd2cfa5e568-osd--block
--28435fba--91c7--4424--99f2--ea86709a87ca
253:2 0 387G 0 lvm
Le vendredi 16 août 2019 à 01:16 +0300, Vitaliy Filippov a écrit :
rbd -p
kube bench kube/bench --io-type write --io-threads 1 --io-
total
10G --io-pattern rand
elapsed: 14 ops: 262144 ops/sec: 17818.16 bytes/sec:
72983201.32
It's a totally unreal number. Something is wrong with the test.
Test it with `fio` please:
fio -ioengine=rbd -name=test -bs=4k -iodepth=1 -rw=randwrite
-runtime=60
-pool=kube -rbdname=bench
Reads are very very slow:
elapsed: 445 ops: 81216 ops/sec: 182.37 bytes/sec:
747006.15
elapsed: 14 ops: 14153 ops/sec: 957.57 bytes/sec:
3922192.15
This is closer to the reality.
7:36 p.m.
Now to go for "apples to apples" either run
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1
-rw=randwrite -runtime=60 -filename=/dev/nvmeXXXXXXXXX
to compare with the single-threaded RBD random write result (the test is
destructive, so use a separate partition without data)
...Or run
fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=write -runtime=60
-pool=kube -rbdname=bench
to compare with your dd's linear write result.
58 single-threaded random iops for NVMes is pretty sad either way. Are
you NVMe's server ones? Do they have capacitors? :) in the response to a
likely question about what they are I'll just post my link here again :)
https://yourcmc.ru/wiki/Ceph_performance
--
Vitaliy Filippov
8:07 p.m.
hello
here on the nvme partition directly
- libaio randwrite /dev/nvme1n1p4 => WRITE: bw=12.1MiB/s (12.7MB/s),
12.1MiB/s-12.1MiB/s (12.7MB/s-12.7MB/s), io=728MiB (763MB), run=60001-
60001msec
- libaio randread /dev/nvme1n1p4 => READ: bw=35.6MiB/s (37.3MB/s),
35.6MiB/s-35.6MiB/s (37.3MB/s-37.3MB/s), io=2134MiB (2237MB),
run=60001-60001msec
here on the rbd
- rbd read : READ: bw=580MiB/s (608MB/s), 580MiB/s-580MiB/s (608MB/s-
608MB/s), io=10.0GiB (10.7GB), run=17668-17668msec (I want this perf !
:) )
- rbd write : WRITE: bw=90.9MiB/s (95.3MB/s), 90.9MiB/s-90.9MiB/s
(95.3MB/s-95.3MB/s), io=5764MiB (6044MB), run=63404-63404msec (I want
this perf ! :) )
here on the mapped rbd
- libaio randwrite on mapped rbd : WRITE: bw=217KiB/s (223kB/s),
217KiB/s-217KiB/s (223kB/s-223kB/s), io=12.7MiB (13.4MB), run=60006-
60006msec
- libaio randread on mapped rbd : READ: bw=589KiB/s (603kB/s),
589KiB/s-589KiB/s (603kB/s-603kB/s), io=34.5MiB (36.2MB), run=60005-
60005msec
here on the mounted fs :
rbd map bench --pool kube --name client.admin
/sbin/mkfs.ext4 /dev/rbd/kube/bench
mount /dev/rbd/kube/bench /mnt/
cd /mnt/
dd if=/dev/zero of=test bs=8192k count=100 oflag=direct
838860800 bytes (839 MB, 800 MiB) copied, 24.5338 s, 34.2 MB/s
raw nvme performance does not looks very great .... but raw
performance of the rbd are great
Once I map it. The performance goes bad. When I use dd in the fs or fio
on the device.
what I don't understand is why the difference between raw rbd and
mapped rbd are so important.
The nvme disk looks to be : Sandisk Corp WD Black 2018/PC SN720 NVMe
SSD
Rand write :
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1
-rw=randwrite -runtime=60 -filename=/dev/nvme1n1p4
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=12.3MiB/s][w=3158 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=177938: Fri Aug 16 17:18:18 2019
write: IOPS=3105, BW=12.1MiB/s (12.7MB/s)(728MiB/60001msec); 0 zone
resets
slat (nsec): min=1545, max=141856, avg=6784.48, stdev=6188.10
clat (nsec): min=688, max=3539.6k, avg=14665.22, stdev=13601.60
lat (usec): min=9, max=3549, avg=21.66, stdev=16.07
clat percentiles (usec):
| 1.00th=[ 8], 5.00th=[ 8], 10.00th=[ 9],
20.00th=[ 9],
| 30.00th=[ 10], 40.00th=[ 10], 50.00th=[ 12],
60.00th=[ 14],
| 70.00th=[ 16], 80.00th=[ 19], 90.00th=[ 26],
95.00th=[ 36],
| 99.00th=[ 49], 99.50th=[ 52], 99.90th=[ 76],
99.95th=[ 85],
| 99.99th=[ 135]
bw ( KiB/s): min=10504, max=13232, per=100.00%, avg=12420.01,
stdev=439.86, samples=119
iops : min= 2626, max= 3308, avg=3105.00, stdev=109.96,
samples=119
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.02%, 4=0.09%, 10=40.60%, 20=43.62%, 50=14.92%
lat (usec) : 100=0.73%, 250=0.02%, 500=0.01%, 750=0.01%
lat (msec) : 4=0.01%
fsync/fdatasync/sync_file_range:
sync (nsec): min=12, max=22490, avg=185.20, stdev=255.75
sync percentiles (nsec):
| 1.00th=[ 28], 5.00th=[ 35], 10.00th=[ 42],
20.00th=[ 59],
| 30.00th=[ 82], 40.00th=[ 109], 50.00th=[ 137],
60.00th=[ 163],
| 70.00th=[ 197], 80.00th=[ 253], 90.00th=[ 390],
95.00th=[ 572],
| 99.00th=[ 804], 99.50th=[ 820], 99.90th=[ 1144], 99.95th=[
1208],
| 99.99th=[15552]
cpu : usr=2.42%, sys=4.91%, ctx=546845, majf=0, minf=12
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
issued rwts:
total=0,186313,0,186313 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=12.1MiB/s (12.7MB/s), 12.1MiB/s-12.1MiB/s (12.7MB/s-
12.7MB/s), io=728MiB (763MB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=0/375662, merge=0/1492, ticks=0/57478, in_queue=59100,
util=97.85%
RBD read
~# fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=read
-runtime=60 -pool=kube -rbdname=bench
test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
4096KiB-4096KiB, ioengine=rbd, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][69.2%][r=1313MiB/s][r=328 IOPS][eta 00m:08s]
test: (groupid=0, jobs=1): err= 0: pid=184386: Fri Aug 16 17:20:45 2019
read: IOPS=144, BW=580MiB/s (608MB/s)(10.0GiB/17668msec)
slat (nsec): min=371, max=72009, avg=5416.87, stdev=3649.19
clat (msec): min=6, max=1438, avg=220.83, stdev=192.27
lat (msec): min=6, max=1438, avg=220.83, stdev=192.27
clat percentiles (msec):
| 1.00th=[ 19], 5.00th=[ 23], 10.00th=[ 25],
20.00th=[ 30],
| 30.00th=[ 33], 40.00th=[ 47], 50.00th=[ 249],
60.00th=[ 288],
| 70.00th=[ 326], 80.00th=[ 372], 90.00th=[ 447],
95.00th=[ 535],
| 99.00th=[ 802], 99.50th=[ 844], 99.90th=[ 936], 99.95th=[
1011],
| 99.99th=[ 1435]
bw ( KiB/s): min=253952, max=4628480, per=93.74%, avg=556353.83,
stdev=851591.15, samples=35
iops : min= 62, max= 1130, avg=135.83, stdev=207.91,
samples=35
lat (msec) : 10=0.08%, 20=2.03%, 50=38.40%, 100=2.15%, 250=7.62%
lat (msec) : 500=43.48%, 750=4.73%, 1000=1.45%
cpu : usr=0.19%, sys=0.24%, ctx=2563, majf=0, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=98.8%,
=64=0.0%
submit : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
=64=0.0%
issued rwts: total=2560,0,0,0
short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=580MiB/s (608MB/s), 580MiB/s-580MiB/s (608MB/s-608MB/s),
io=10.0GiB (10.7GB), run=17668-17668msec
Disk stats (read/write):
md2: ios=1/1405, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=1332/3191, aggrmerge=5911/5187, aggrticks=1031/1257,
aggrin_queue=15480, aggrutil=91.95%
nvme1n1: ios=0/1183, merge=0/450, ticks=0/141, in_queue=12452,
util=69.94%
nvme0n1: ios=2665/5199, merge=11822/9925, ticks=2062/2373,
in_queue=18508, util=91.95%
RBD write
fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=write
-runtime=60 -pool=kube -rbdname=bench
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
4096KiB-4096KiB, ioengine=rbd, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][55.3%][eta 00m:51s]
test: (groupid=0, jobs=1): err= 0: pid=181899: Fri Aug 16 17:20:19 2019
write: IOPS=22, BW=90.9MiB/s (95.3MB/s)(5764MiB/63404msec); 0 zone
resets
slat (usec): min=366, max=4889, avg=802.71, stdev=414.92
clat (msec): min=189, max=7308, avg=1404.62, stdev=1278.48
lat (msec): min=189, max=7308, avg=1405.43, stdev=1278.47
clat percentiles (msec):
| 1.00th=[ 292], 5.00th=[ 376], 10.00th=[ 435],
20.00th=[ 550],
| 30.00th=[ 676], 40.00th=[ 776], 50.00th=[ 877], 60.00th=[
1083],
| 70.00th=[ 1351], 80.00th=[ 1921], 90.00th=[ 3473], 95.00th=[
4597],
| 99.00th=[ 5470], 99.50th=[ 6007], 99.90th=[ 6745], 99.95th=[
7282],
| 99.99th=[ 7282]
bw ( KiB/s): min= 8192, max=155648, per=100.00%, avg=95454.68,
stdev=30757.07, samples=121
iops : min= 2, max= 38, avg=23.27, stdev= 7.52,
samples=121
lat (msec) : 250=0.07%, 500=15.82%, 750=22.21%, 1000=18.74%
cpu : usr=1.71%, sys=0.15%, ctx=597, majf=0, minf=45137
IO depths : 1=0.1%, 2=0.1%, 4=0.3%, 8=0.6%, 16=1.1%, 32=97.8%,
=64=0.0%
submit : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
=64=0.0%
issued rwts: total=0,1441,0,0
short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=90.9MiB/s (95.3MB/s), 90.9MiB/s-90.9MiB/s (95.3MB/s-
95.3MB/s), io=5764MiB (6044MB), run=63404-63404msec
Disk stats (read/write):
md2: ios=0/5119, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=30/16383, aggrmerge=20/28021, aggrticks=5/18949,
aggrin_queue=63466, aggrutil=90.70%
nvme1n1: ios=0/4230, merge=0/1595, ticks=0/1835, in_queue=42440,
util=65.48%
nvme0n1: ios=60/28536, merge=41/54447, ticks=10/36063,
in_queue=84492, util=90.70%
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1
-rw=randread -runtime=60 -filename=/dev/nvme1n1p4
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=36.3MiB/s][r=9286 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=208060: Fri Aug 16 17:33:01 2019
read: IOPS=9103, BW=35.6MiB/s (37.3MB/s)(2134MiB/60001msec)
slat (nsec): min=1384, max=244751, avg=6077.86, stdev=5529.26
clat (usec): min=3, max=8311, avg=101.98, stdev=42.25
lat (usec): min=34, max=8341, avg=108.28, stdev=43.81
clat percentiles (usec):
| 1.00th=[ 73], 5.00th=[ 83], 10.00th=[ 85],
20.00th=[ 90],
| 30.00th=[ 92], 40.00th=[ 94], 50.00th=[ 96],
60.00th=[ 100],
| 70.00th=[ 104], 80.00th=[ 112], 90.00th=[ 121],
95.00th=[ 141],
| 99.00th=[ 182], 99.50th=[ 198], 99.90th=[ 253],
99.95th=[ 297],
| 99.99th=[ 2147]
bw ( KiB/s): min=33088, max=40928, per=99.96%, avg=36400.00,
stdev=1499.29, samples=119
iops : min= 8272, max=10232, avg=9099.98, stdev=374.82,
samples=119
lat (usec) : 4=0.01%, 10=0.01%, 50=0.53%, 100=59.74%, 250=39.62%
lat (usec) : 500=0.08%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=4.80%, sys=8.33%, ctx=546238, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
issued rwts: total=546228,0,0,0
short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=35.6MiB/s (37.3MB/s), 35.6MiB/s-35.6MiB/s (37.3MB/s-
37.3MB/s), io=2134MiB (2237MB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=545111/3564, merge=0/1430, ticks=54388/943,
in_queue=60924, util=100.00%
Le vendredi 16 août 2019 à 18:06 +0300, vitalif(a)yourcmc.ru a écrit :
Now to go for "apples to apples" either run
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1
-rw=randwrite -runtime=60 -filename=/dev/nvmeXXXXXXXXX
to compare with the single-threaded RBD random write result (the test
is
destructive, so use a separate partition without data)
...Or run
fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=write
-runtime=60
-pool=kube -rbdname=bench
to compare with your dd's linear write result.
58 single-threaded random iops for NVMes is pretty sad either way.
Are
you NVMe's server ones? Do they have capacitors? :) in the response
to a
likely question about what they are I'll just post my link here again
:)
https://yourcmc.ru/wiki/Ceph_performance
8:23 p.m.
Ok I read your link. My ssds are bad. They got capacitors ... I don't
choose them. They come with the hardware I rent. Perhaps it will be
better to switch to hdd. I cannot even but journal on them ... bad news
:(
Le vendredi 16 août 2019 à 17:37 +0200, Olivier AUDRY a écrit :
hello
here on the nvme partition directly
- libaio randwrite /dev/nvme1n1p4 => WRITE: bw=12.1MiB/s (12.7MB/s),
12.1MiB/s-12.1MiB/s (12.7MB/s-12.7MB/s), io=728MiB (763MB),
run=60001-60001msec
- libaio randread /dev/nvme1n1p4 => READ: bw=35.6MiB/s (37.3MB/s),
35.6MiB/s-35.6MiB/s (37.3MB/s-37.3MB/s), io=2134MiB (2237MB),
run=60001-60001msec
here on the rbd
- rbd read : READ: bw=580MiB/s (608MB/s), 580MiB/s-580MiB/s (608MB/s-
608MB/s), io=10.0GiB (10.7GB), run=17668-17668msec (I want this perf
! :) )
- rbd write : WRITE: bw=90.9MiB/s (95.3MB/s), 90.9MiB/s-90.9MiB/s
(95.3MB/s-95.3MB/s), io=5764MiB (6044MB), run=63404-63404msec (I want
this perf ! :) )
here on the mapped rbd
- libaio randwrite on mapped rbd : WRITE: bw=217KiB/s (223kB/s),
217KiB/s-217KiB/s (223kB/s-223kB/s), io=12.7MiB (13.4MB), run=60006-
60006msec
- libaio randread on mapped rbd : READ: bw=589KiB/s (603kB/s),
589KiB/s-589KiB/s (603kB/s-603kB/s), io=34.5MiB (36.2MB), run=60005-
60005msec
here on the mounted fs :
rbd map bench --pool kube --name client.admin
/sbin/mkfs.ext4 /dev/rbd/kube/bench
mount /dev/rbd/kube/bench /mnt/
cd /mnt/
dd if=/dev/zero of=test bs=8192k count=100 oflag=direct
838860800 bytes (839 MB, 800 MiB) copied, 24.5338 s, 34.2 MB/s
raw nvme performance does not looks very great .... but raw
performance of the rbd are great
Once I map it. The performance goes bad. When I use dd in the fs or
fio on the device.
what I don't understand is why the difference between raw rbd and
mapped rbd are so important.
The nvme disk looks to be : Sandisk Corp WD Black 2018/PC SN720 NVMe
SSD
Rand write :
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1
-rw=randwrite -runtime=60 -filename=/dev/nvme1n1p4
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=12.3MiB/s][w=3158 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=177938: Fri Aug 16 17:18:18
2019
write: IOPS=3105, BW=12.1MiB/s (12.7MB/s)(728MiB/60001msec); 0 zone
resets
slat (nsec): min=1545, max=141856, avg=6784.48, stdev=6188.10
clat (nsec): min=688, max=3539.6k, avg=14665.22, stdev=13601.60
lat (usec): min=9, max=3549, avg=21.66, stdev=16.07
clat percentiles (usec):
| 1.00th=[ 8], 5.00th=[ 8], 10.00th=[ 9],
20.00th=[ 9],
| 30.00th=[ 10], 40.00th=[ 10], 50.00th=[ 12],
60.00th=[ 14],
| 70.00th=[ 16], 80.00th=[ 19], 90.00th=[ 26],
95.00th=[ 36],
| 99.00th=[ 49], 99.50th=[ 52], 99.90th=[ 76],
99.95th=[ 85],
| 99.99th=[ 135]
bw ( KiB/s): min=10504, max=13232, per=100.00%, avg=12420.01,
stdev=439.86, samples=119
iops : min= 2626, max= 3308, avg=3105.00, stdev=109.96,
samples=119
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.02%, 4=0.09%, 10=40.60%, 20=43.62%, 50=14.92%
lat (usec) : 100=0.73%, 250=0.02%, 500=0.01%, 750=0.01%
lat (msec) : 4=0.01%
fsync/fdatasync/sync_file_range:
sync (nsec): min=12, max=22490, avg=185.20, stdev=255.75
sync percentiles (nsec):
| 1.00th=[ 28], 5.00th=[ 35], 10.00th=[ 42],
20.00th=[ 59],
| 30.00th=[ 82], 40.00th=[ 109], 50.00th=[ 137],
60.00th=[ 163],
| 70.00th=[ 197], 80.00th=[ 253], 90.00th=[ 390],
95.00th=[ 572],
| 99.00th=[ 804], 99.50th=[ 820], 99.90th=[ 1144], 99.95th=[
1208],
| 99.99th=[15552]
cpu : usr=2.42%, sys=4.91%, ctx=546845, majf=0, minf=12
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
issued rwts:
total=0,186313,0,186313 short=0,0,0,0
dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=12.1MiB/s (12.7MB/s), 12.1MiB/s-12.1MiB/s (12.7MB/s-
12.7MB/s), io=728MiB (763MB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=0/375662, merge=0/1492, ticks=0/57478, in_queue=59100,
util=97.85%
RBD read
~# fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=read
-runtime=60 -pool=kube -rbdname=bench
test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB,
(T) 4096KiB-4096KiB, ioengine=rbd, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][69.2%][r=1313MiB/s][r=328 IOPS][eta 00m:08s]
test: (groupid=0, jobs=1): err= 0: pid=184386: Fri Aug 16 17:20:45
2019
read: IOPS=144, BW=580MiB/s (608MB/s)(10.0GiB/17668msec)
slat (nsec): min=371, max=72009, avg=5416.87, stdev=3649.19
clat (msec): min=6, max=1438, avg=220.83, stdev=192.27
lat (msec): min=6, max=1438, avg=220.83, stdev=192.27
clat percentiles (msec):
| 1.00th=[ 19], 5.00th=[ 23], 10.00th=[ 25],
20.00th=[ 30],
| 30.00th=[ 33], 40.00th=[ 47], 50.00th=[ 249],
60.00th=[ 288],
| 70.00th=[ 326], 80.00th=[ 372], 90.00th=[ 447],
95.00th=[ 535],
| 99.00th=[ 802], 99.50th=[ 844], 99.90th=[ 936], 99.95th=[
1011],
| 99.99th=[ 1435]
bw ( KiB/s): min=253952, max=4628480, per=93.74%, avg=556353.83,
stdev=851591.15, samples=35
iops : min= 62, max= 1130, avg=135.83, stdev=207.91,
samples=35
lat (msec) : 10=0.08%, 20=2.03%, 50=38.40%, 100=2.15%, 250=7.62%
lat (msec) : 500=43.48%, 750=4.73%, 1000=1.45%
cpu : usr=0.19%, sys=0.24%, ctx=2563, majf=0, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=98.8%,
=64=0.0%
submit : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
=64=0.0%
issued rwts: total=2560,0,0,0
short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=580MiB/s (608MB/s), 580MiB/s-580MiB/s (608MB/s-608MB/s),
io=10.0GiB (10.7GB), run=17668-17668msec
Disk stats (read/write):
md2: ios=1/1405, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=1332/3191, aggrmerge=5911/5187, aggrticks=1031/1257,
aggrin_queue=15480, aggrutil=91.95%
nvme1n1: ios=0/1183, merge=0/450, ticks=0/141, in_queue=12452,
util=69.94%
nvme0n1: ios=2665/5199, merge=11822/9925, ticks=2062/2373,
in_queue=18508, util=91.95%
RBD write
fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=write
-runtime=60 -pool=kube -rbdname=bench
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB,
(T) 4096KiB-4096KiB, ioengine=rbd, iodepth=32
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][55.3%][eta 00m:51s]
test: (groupid=0, jobs=1): err= 0: pid=181899: Fri Aug 16 17:20:19
2019
write: IOPS=22, BW=90.9MiB/s (95.3MB/s)(5764MiB/63404msec); 0 zone
resets
slat (usec): min=366, max=4889, avg=802.71, stdev=414.92
clat (msec): min=189, max=7308, avg=1404.62, stdev=1278.48
lat (msec): min=189, max=7308, avg=1405.43, stdev=1278.47
clat percentiles (msec):
| 1.00th=[ 292], 5.00th=[ 376], 10.00th=[ 435],
20.00th=[ 550],
| 30.00th=[ 676], 40.00th=[ 776], 50.00th=[ 877], 60.00th=[
1083],
| 70.00th=[ 1351], 80.00th=[ 1921], 90.00th=[ 3473], 95.00th=[
4597],
| 99.00th=[ 5470], 99.50th=[ 6007], 99.90th=[ 6745], 99.95th=[
7282],
| 99.99th=[ 7282]
bw ( KiB/s): min= 8192, max=155648, per=100.00%, avg=95454.68,
stdev=30757.07, samples=121
iops : min= 2, max= 38, avg=23.27, stdev= 7.52,
samples=121
lat (msec) : 250=0.07%, 500=15.82%, 750=22.21%, 1000=18.74%
cpu : usr=1.71%, sys=0.15%, ctx=597, majf=0, minf=45137
IO depths : 1=0.1%, 2=0.1%, 4=0.3%, 8=0.6%, 16=1.1%, 32=97.8%,
=64=0.0%
submit : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
=64=0.0%
issued rwts: total=0,1441,0,0
short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=90.9MiB/s (95.3MB/s), 90.9MiB/s-90.9MiB/s (95.3MB/s-
95.3MB/s), io=5764MiB (6044MB), run=63404-63404msec
Disk stats (read/write):
md2: ios=0/5119, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=30/16383, aggrmerge=20/28021, aggrticks=5/18949,
aggrin_queue=63466, aggrutil=90.70%
nvme1n1: ios=0/4230, merge=0/1595, ticks=0/1835, in_queue=42440,
util=65.48%
nvme0n1: ios=60/28536, merge=41/54447, ticks=10/36063,
in_queue=84492, util=90.70%
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1
-rw=randread -runtime=60 -filename=/dev/nvme1n1p4
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=36.3MiB/s][r=9286 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=208060: Fri Aug 16 17:33:01
2019
read: IOPS=9103, BW=35.6MiB/s (37.3MB/s)(2134MiB/60001msec)
slat (nsec): min=1384, max=244751, avg=6077.86, stdev=5529.26
clat (usec): min=3, max=8311, avg=101.98, stdev=42.25
lat (usec): min=34, max=8341, avg=108.28, stdev=43.81
clat percentiles (usec):
| 1.00th=[ 73], 5.00th=[ 83], 10.00th=[ 85],
20.00th=[ 90],
| 30.00th=[ 92], 40.00th=[ 94], 50.00th=[ 96],
60.00th=[ 100],
| 70.00th=[ 104], 80.00th=[ 112], 90.00th=[ 121],
95.00th=[ 141],
| 99.00th=[ 182], 99.50th=[ 198], 99.90th=[ 253],
99.95th=[ 297],
| 99.99th=[ 2147]
bw ( KiB/s): min=33088, max=40928, per=99.96%, avg=36400.00,
stdev=1499.29, samples=119
iops : min= 8272, max=10232, avg=9099.98, stdev=374.82,
samples=119
lat (usec) : 4=0.01%, 10=0.01%, 50=0.53%, 100=59.74%, 250=39.62%
lat (usec) : 500=0.08%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=4.80%, sys=8.33%, ctx=546238, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
issued rwts:
total=546228,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=35.6MiB/s (37.3MB/s), 35.6MiB/s-35.6MiB/s (37.3MB/s-
37.3MB/s), io=2134MiB (2237MB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=545111/3564, merge=0/1430, ticks=54388/943,
in_queue=60924, util=100.00%
Le vendredi 16 août 2019 à 18:06 +0300, vitalif(a)yourcmc.ru a écrit :
Now to go for "apples to apples" either
run
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1
-fsync=1
-rw=randwrite -runtime=60 -filename=/dev/nvmeXXXXXXXXX
to compare with the single-threaded RBD random write result (the
test is
destructive, so use a separate partition without data)
...Or run
fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=write
-runtime=60
-pool=kube -rbdname=bench
to compare with your dd's linear write result.
58 single-threaded random iops for NVMes is pretty sad either way.
Are
you NVMe's server ones? Do they have capacitors? :) in the response
to a
likely question about what they are I'll just post my link here
again :)
https://yourcmc.ru/wiki/Ceph_performance
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:34 p.m.
- libaio randwrite
- libaio randread
- libaio randwrite on mapped rbd
- libaio randread on mapped rbd
- rbd read
- rbd write
recheck RBD with RAND READ / RAND WRITE
you're again comparing RANDOM and NON-RANDOM I/O
your SSDs aren't that bad, 3000 single-thread iops isn't the worst
possible performance
10:58 p.m.
fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=randwrite
-runtime=60 -pool=kube -rbdname=bench
WRITE: bw=89.6MiB/s (93.9MB/s), 89.6MiB/s-89.6MiB/s (93.9MB/s-
93.9MB/s), io=5548MiB (5817MB), run=61935-61935msec
fio -ioengine=rbd -name=test -bs=4M -iodepth=32 -rw=randread
-runtime=60 -pool=kube -rbdname=bench
READ: bw=139MiB/s (146MB/s), 139MiB/s-139MiB/s (146MB/s-146MB/s),
io=8404MiB (8812MB), run=60443-60443msec
this is great. I don't get why I can't get the same in the mapped rbd.
Le vendredi 16 août 2019 à 21:04 +0300, vitalif(a)yourcmc.ru a écrit :
- libaio
randwrite
- libaio randread
- libaio randwrite on mapped rbd
- libaio randread on mapped rbd
- rbd read
- rbd write
recheck RBD with RAND READ / RAND WRITE
you're again comparing RANDOM and NON-RANDOM I/O
your SSDs aren't that bad, 3000 single-thread iops isn't the worst
possible performance
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:04 p.m.
And once more you're checking random I/O with 4 MB !!! block size.
Now recheck it with bs=4k.
11:47 p.m.
RBD
fio -ioengine=rbd -name=test -bs=4k -iodepth=32 -rw=randread
-runtime=60 -pool=kube -rbdname=bench
READ: bw=21.8MiB/s (22.8MB/s), 21.8MiB/s-21.8MiB/s (22.8MB/s-22.8MB/s),
io=1308MiB (1371MB), run=60011-60011msec
fio -ioengine=rbd -name=test -bs=4k -iodepth=32 -rw=randwrite
-runtime=60 -pool=kube -rbdname=bench
WRITE: bw=5968KiB/s (6111kB/s), 5968KiB/s-5968KiB/s (6111kB/s-
6111kB/s), io=350MiB (367MB), run=60022-60022msec
mapped rbd :
fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randread
-runtime=60 -filename=/dev/rbd/kube/bench
READ: bw=785KiB/s (804kB/s), 785KiB/s-785KiB/s (804kB/s-804kB/s),
io=46.0MiB (48.3MB), run=60008-60008msec
old hardware raw disk:
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1
-rw=randwrite -runtime=60 -filename=/dev/sda4
WRITE: bw=107MiB/s (112MB/s), 107MiB/s-107MiB/s (112MB/s-112MB/s),
io=6401MiB (6712MB), run=60001-60001msec
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1
-rw=randread -runtime=60 -filename=/dev/sda4
READ: bw=36.8MiB/s (38.6MB/s), 36.8MiB/s-36.8MiB/s (38.6MB/s-38.6MB/s),
io=2208MiB (2315MB), run=60001-60001msec
new hardware raw disk :
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1
-rw=randwrite -runtime=60 -filename=/dev/nvme1n1p4
WRITE: bw=12.1MiB/s (12.7MB/s), 12.1MiB/s-12.1MiB/s (12.7MB/s-
12.7MB/s), io=728MiB (763MB), run=60001-60001msec
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1
-rw=randread -runtime=60 -filename=/dev/nvme1n1p4
READ: bw=35.6MiB/s (37.3MB/s), 35.6MiB/s-35.6MiB/s (37.3MB/s-37.3MB/s),
io=2134MiB (2237MB), run=60001-60001msec
Le vendredi 16 août 2019 à 21:34 +0300, vitalif(a)yourcmc.ru a écrit :
And once more you're checking random I/O with 4 MB
!!! block size.
Now recheck it with bs=4k.
12:42 a.m.
on a new ceph cluster with the same software and config (ansible) on
the old hardware. 2 replica, 1 host, 4 osd.
RBD
fio -ioengine=rbd -name=test -bs=4k -iodepth=32 -rw=randread
-runtime=60 -pool=kube -rbdname=bench
READ: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s),
io=7189MiB (7538MB), run=60001-60001msec
fio -ioengine=rbd -name=test -bs=4k -iodepth=32 -rw=randwrite
-runtime=60 -pool=kube -rbdname=bench
WRITE: bw=42.0MiB/s (44.1MB/s), 42.0MiB/s-42.0MiB/s (44.1MB/s-
44.1MB/s), io=2522MiB (2645MB), run=60004-60004msec
MAPPED RBD
fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randread
-direct=1 -runtime=60 -filename=/dev/rbd/kube/bench
READ: bw=184MiB/s (193MB/s), 184MiB/s-184MiB/s (193MB/s-193MB/s),
io=10.0GiB (10.7GB), run=55613-55613msec
fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randwrite
-direct=1 -runtime=60 -filename=/dev/rbd/kube/bench
WRITE: bw=46.9MiB/s (49.2MB/s), 46.9MiB/s-46.9MiB/s (49.2MB/s-
49.2MB/s), io=2814MiB (2950MB), run=60002-60002msec
looks much better old the old one
on the cluster with the new hardware. 2 replica 3 host 6 osd.
fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randread
-direct=1 -runtime=60 -filename=/dev/rbd/kube/bench
READ: bw=31.1MiB/s (32.6MB/s), 31.1MiB/s-31.1MiB/s (32.6MB/s-32.6MB/s),
io=1866MiB (1957MB), run=60010-60010msec
fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randwrite
-direct=1 -runtime=60 -filename=/dev/rbd/kube/bench
WRITE: bw=10.5MiB/s (11.0MB/s), 10.5MiB/s-10.5MiB/s (11.0MB/s-
11.0MB/s), io=631MiB (662MB), run=60021-60021msec
=> New hardware : 32.6MB/s READ / 10.5MiB WRITE
=> Old hardware : 184MiB/s READ / 46.9MiB WRITE
No discussion ? I suppose I will keep the old hardware. What do you
think ? :D
Le vendredi 16 août 2019 à 21:17 +0200, Olivier AUDRY a écrit :
RBD
fio -ioengine=rbd -name=test -bs=4k -iodepth=32 -rw=randread
-runtime=60 -pool=kube -rbdname=bench
READ: bw=21.8MiB/s (22.8MB/s), 21.8MiB/s-21.8MiB/s (22.8MB/s-
22.8MB/s), io=1308MiB (1371MB), run=60011-60011msec
fio -ioengine=rbd -name=test -bs=4k -iodepth=32 -rw=randwrite
-runtime=60 -pool=kube -rbdname=bench
WRITE: bw=5968KiB/s (6111kB/s), 5968KiB/s-5968KiB/s (6111kB/s-
6111kB/s), io=350MiB (367MB), run=60022-60022msec
mapped rbd :
fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randread
-runtime=60 -filename=/dev/rbd/kube/bench
READ: bw=785KiB/s (804kB/s), 785KiB/s-785KiB/s (804kB/s-804kB/s),
io=46.0MiB (48.3MB), run=60008-60008msec
old hardware raw disk:
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1
-rw=randwrite -runtime=60 -filename=/dev/sda4
WRITE: bw=107MiB/s (112MB/s), 107MiB/s-107MiB/s (112MB/s-112MB/s),
io=6401MiB (6712MB), run=60001-60001msec
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1
-rw=randread -runtime=60 -filename=/dev/sda4
READ: bw=36.8MiB/s (38.6MB/s), 36.8MiB/s-36.8MiB/s (38.6MB/s-
38.6MB/s), io=2208MiB (2315MB), run=60001-60001msec
new hardware raw disk :
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1
-rw=randwrite -runtime=60 -filename=/dev/nvme1n1p4
WRITE: bw=12.1MiB/s (12.7MB/s), 12.1MiB/s-12.1MiB/s (12.7MB/s-
12.7MB/s), io=728MiB (763MB), run=60001-60001msec
fio -ioengine=libaio -name=test -bs=4k -iodepth=1 -direct=1 -fsync=1
-rw=randread -runtime=60 -filename=/dev/nvme1n1p4
READ: bw=35.6MiB/s (37.3MB/s), 35.6MiB/s-35.6MiB/s (37.3MB/s-
37.3MB/s), io=2134MiB (2237MB), run=60001-60001msec
Le vendredi 16 août 2019 à 21:34 +0300, vitalif(a)yourcmc.ru a écrit :
And once more you're checking random I/O with
4 MB !!! block size.
Now recheck it with bs=4k.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:25 a.m.
on a new ceph cluster with the same software and
config (ansible) on
the old hardware. 2 replica, 1 host, 4 osd.
=> New hardware : 32.6MB/s READ / 10.5MiB WRITE
=> Old hardware : 184MiB/s READ / 46.9MiB WRITE
No discussion ? I suppose I will keep the old hardware. What do you
think ? :D
In fact I don't really believe in 184 MB/s random reads with Ceph with 4
OSDs, it's a very cool result if it's true.
Does the "new cluster on the old hardware" consist of only 1 host? Did
you test reads before you actually wrote anything into the image so it
was empty and reads were fast because of that?
3:46 a.m.
Write and read with 2 hosts 4 osd :
mkfs.ext4 /dev/rbd/kube/bench
mount /dev/rbd/kube/bench /mnt/
dd if=/dev/zero of=test bs=8192k count=1000 oflag=direct
8388608000 bytes (8.4 GB, 7.8 GiB) copied, 117.541 s, 71.4 MB/s
fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randwrite
-direct=1 -runtime=60 -filename=/dev/rbd/kube/bench
WRITE: bw=45.3MiB/s (47.5MB/s), 45.3MiB/s-45.3MiB/s (47.5MB/s-
47.5MB/s), io=2718MiB (2850MB), run=60003-60003msec
fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randread
-direct=1 -runtime=60 -filename=/dev/rbd/kube/bench
READ: bw=187MiB/s (197MB/s), 187MiB/s-187MiB/s (197MB/s-197MB/s),
io=10.0GiB (10.7GB), run=54636-54636msec
pgbench before : 10 transaction per second
pgbench after : 355 transaction per second
So yes it's better. SSD are INTEL SSDSC2BB48 0370.
Le samedi 17 août 2019 à 01:55 +0300, vitalif(a)yourcmc.ru a écrit :
on a new ceph
cluster with the same software and config (ansible)
on
the old hardware. 2 replica, 1 host, 4 osd.
=> New hardware : 32.6MB/s READ / 10.5MiB WRITE
=> Old hardware : 184MiB/s READ / 46.9MiB WRITE
No discussion ? I suppose I will keep the old hardware. What do you
think ? :D
In fact I don't really believe in 184 MB/s random reads with Ceph
with 4
OSDs, it's a very cool result if it's true.
Does the "new cluster on the old hardware" consist of only 1 host?
Did
you test reads before you actually wrote anything into the image so
it
was empty and reads were fast because of that?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
4:40 a.m.
This probably muddies the water. Note Active cluster with around 22
read/write IOPS and 200kB read/write
A CephFS mounted with 3 hosts 6 osd per host with 8G public and 10G
private networking for Ceph.
No SSDs and mostly WD Red 1T 2.5" drives some are HGST 1T 7200.
root@blade7:~# fio -ioengine=libaio -name=test -bs=4k -iodepth=32
-rw=randwrite -direct=1 -runtime=60 -filename=/mnt/pve/cephfs/test
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: you need to specify size=
fio: pid=0, err=22/file:filesetup.c:952, func=total_file_size,
error=Invalid argument
Run status group 0 (all jobs):
root@blade7:~# fio -ioengine=libaio -name=test -bs=4k -iodepth=32
-rw=randwrite -direct=1 -runtime=60 -size=10G -filename=/mnt/pve/cephfs/test
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=0): [f(1)][100.0%][w=580KiB/s][w=145 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=3561674: Sat Aug 17 09:20:22 2019
write: IOPS=2262, BW=9051KiB/s (9268kB/s)(538MiB/60845msec); 0 zone resets
slat (usec): min=8, max=35648, avg=40.01, stdev=97.51
clat (usec): min=954, max=2854.3k, avg=14090.15, stdev=100194.83
lat (usec): min=994, max=2854.3k, avg=14130.65, stdev=100195.40
clat percentiles (usec):
| 1.00th=[ 1254], 5.00th=[ 1450], 10.00th=[ 1582],
| 20.00th=[ 1795], 30.00th=[ 2008], 40.00th=[ 2245],
| 50.00th=[ 2540], 60.00th=[ 2933], 70.00th=[ 3392],
| 80.00th=[ 4228], 90.00th=[ 7767], 95.00th=[ 35914],
| 99.00th=[ 254804], 99.50th=[ 616563], 99.90th=[1652556],
| 99.95th=[2122318], 99.99th=[2600469]
bw ( KiB/s): min= 48, max=44408, per=100.00%, avg=10387.54,
stdev=10384.94, samples=106
iops : min= 12, max=11102, avg=2596.88, stdev=2596.23,
samples=106
lat (usec) : 1000=0.01%
lat (msec) : 2=29.82%, 4=47.95%, 10=14.23%, 20=2.43%, 50=1.34%
lat (msec) : 100=2.68%, 250=0.53%, 500=0.40%, 750=0.20%, 1000=0.14%
cpu : usr=1.45%, sys=6.36%, ctx=151946, majf=0, minf=280
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
=64=0.0%
issued rwts: total=0,137674,0,0
short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=9051KiB/s (9268kB/s), 9051KiB/s-9051KiB/s
(9268kB/s-9268kB/s), io=538MiB (564MB), run=60845-60845msec
This is on the same system with a RBD mapped file system
root@blade7:/mnt# fio -ioengine=libaio -name=test -bs=4k -iodepth=32
-rw=randwrite -direct=1 -runtime=60 -size=10G -filename=/mnt/image0/test
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [w(1)][4.5%][w=4KiB/s][w=1 IOPS][eta 21m:30s]
test: (groupid=0, jobs=1): err= 0: pid=3567399: Sat Aug 17 09:38:55 2019
write: IOPS=1935, BW=7744KiB/s (7930kB/s)(462MiB/61143msec); 0 zone resets
slat (usec): min=9, max=700161, avg=65.17, stdev=2092.54
clat (usec): min=954, max=2578.6k, avg=16457.67, stdev=109995.03
lat (usec): min=1021, max=2578.6k, avg=16523.42, stdev=110014.91
clat percentiles (usec):
| 1.00th=[ 1254], 5.00th=[ 1434], 10.00th=[ 1549],
| 20.00th=[ 1745], 30.00th=[ 1909], 40.00th=[ 2114],
| 50.00th=[ 2376], 60.00th=[ 2704], 70.00th=[ 3228],
| 80.00th=[ 4080], 90.00th=[ 8717], 95.00th=[ 53216],
| 99.00th=[ 291505], 99.50th=[ 675283], 99.90th=[1669333],
| 99.95th=[2231370], 99.99th=[2365588]
bw ( KiB/s): min= 8, max=35968, per=100.00%, avg=9015.64,
stdev=8402.84, samples=105
iops : min= 2, max= 8992, avg=2253.90, stdev=2100.72,
samples=105
lat (usec) : 1000=0.01%
lat (msec) : 2=34.85%, 4=44.49%, 10=11.54%, 20=1.84%, 50=1.81%
lat (msec) : 100=3.27%, 250=1.13%, 500=0.42%, 750=0.19%, 1000=0.08%
cpu : usr=1.42%, sys=6.63%, ctx=123309, majf=0, minf=283
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
=64=0.0%
issued rwts: total=0,118371,0,0
short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=7744KiB/s (7930kB/s), 7744KiB/s-7744KiB/s
(7930kB/s-7930kB/s), io=462MiB (485MB), run=61143-61143msec
Disk stats (read/write):
rbd0: ios=0/118670, merge=0/9674, ticks=0/1894238, in_queue=1651008,
util=33.33%
On 17/8/19 8:46 am, Olivier AUDRY wrote:
Write and read with 2 hosts 4 osd :
mkfs.ext4 /dev/rbd/kube/bench
mount /dev/rbd/kube/bench /mnt/
dd if=/dev/zero of=test bs=8192k count=1000 oflag=direct
8388608000 bytes (8.4 GB, 7.8 GiB) copied, 117.541 s, 71.4 MB/s
fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randwrite
-direct=1 -runtime=60 -filename=/dev/rbd/kube/bench
WRITE: bw=45.3MiB/s (47.5MB/s), 45.3MiB/s-45.3MiB/s (47.5MB/s-
47.5MB/s), io=2718MiB (2850MB), run=60003-60003msec
fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randread
-direct=1 -runtime=60 -filename=/dev/rbd/kube/bench
READ: bw=187MiB/s (197MB/s), 187MiB/s-187MiB/s (197MB/s-197MB/s),
io=10.0GiB (10.7GB), run=54636-54636msec
pgbench before : 10 transaction per second
pgbench after : 355 transaction per second
So yes it's better. SSD are INTEL SSDSC2BB48 0370.
Le samedi 17 août 2019 à 01:55 +0300, vitalif(a)yourcmc.ru a écrit :
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
on a new
ceph cluster with the same software and config (ansible)
on
the old hardware. 2 replica, 1 host, 4 osd.
=> New hardware : 32.6MB/s READ / 10.5MiB WRITE
=> Old hardware : 184MiB/s READ / 46.9MiB WRITE
No discussion ? I suppose I will keep the old hardware. What do you
think ? :D
In fact I don't really believe in 184 MB/s random reads with Ceph
with 4
OSDs, it's a very cool result if it's true.
Does the "new cluster on the old hardware" consist of only 1 host?
Did
you test reads before you actually wrote anything into the image so
it
was empty and reads were fast because of that?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1708
days inactive
1711
days old
18 comments
6 participants
participants (6)
-
Ilya Dryomov -
Mike O'Connor -
Olivier AUDRY -
Paul Emmerich -
vitalif＠yourcmc.ru -
Vitaliy Filippov