This probably muddies the water. Note Active cluster with around 22
read/write IOPS and 200kB read/write
A CephFS mounted with 3 hosts 6 osd per host with 8G public and 10G
private networking for Ceph.
No SSDs and mostly WD Red 1T 2.5" drives some are HGST 1T 7200.
root@blade7:~# fio -ioengine=libaio -name=test -bs=4k -iodepth=32
-rw=randwrite -direct=1 -runtime=60 -filename=/mnt/pve/cephfs/test
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: you need to specify size=
fio: pid=0, err=22/file:filesetup.c:952, func=total_file_size,
error=Invalid argument
Run status group 0 (all jobs):
root@blade7:~# fio -ioengine=libaio -name=test -bs=4k -iodepth=32
-rw=randwrite -direct=1 -runtime=60 -size=10G -filename=/mnt/pve/cephfs/test
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=0): [f(1)][100.0%][w=580KiB/s][w=145 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=3561674: Sat Aug 17 09:20:22 2019
write: IOPS=2262, BW=9051KiB/s (9268kB/s)(538MiB/60845msec); 0 zone resets
slat (usec): min=8, max=35648, avg=40.01, stdev=97.51
clat (usec): min=954, max=2854.3k, avg=14090.15, stdev=100194.83
lat (usec): min=994, max=2854.3k, avg=14130.65, stdev=100195.40
clat percentiles (usec):
| 1.00th=[ 1254], 5.00th=[ 1450], 10.00th=[ 1582],
| 20.00th=[ 1795], 30.00th=[ 2008], 40.00th=[ 2245],
| 50.00th=[ 2540], 60.00th=[ 2933], 70.00th=[ 3392],
| 80.00th=[ 4228], 90.00th=[ 7767], 95.00th=[ 35914],
| 99.00th=[ 254804], 99.50th=[ 616563], 99.90th=[1652556],
| 99.95th=[2122318], 99.99th=[2600469]
bw ( KiB/s): min= 48, max=44408, per=100.00%, avg=10387.54,
stdev=10384.94, samples=106
iops : min= 12, max=11102, avg=2596.88, stdev=2596.23,
samples=106
lat (usec) : 1000=0.01%
lat (msec) : 2=29.82%, 4=47.95%, 10=14.23%, 20=2.43%, 50=1.34%
lat (msec) : 100=2.68%, 250=0.53%, 500=0.40%, 750=0.20%, 1000=0.14%
cpu : usr=1.45%, sys=6.36%, ctx=151946, majf=0, minf=280
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
=64=0.0%
issued rwts: total=0,137674,0,0
short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=9051KiB/s (9268kB/s), 9051KiB/s-9051KiB/s
(9268kB/s-9268kB/s), io=538MiB (564MB), run=60845-60845msec
This is on the same system with a RBD mapped file system
root@blade7:/mnt# fio -ioengine=libaio -name=test -bs=4k -iodepth=32
-rw=randwrite -direct=1 -runtime=60 -size=10G -filename=/mnt/image0/test
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [w(1)][4.5%][w=4KiB/s][w=1 IOPS][eta 21m:30s]
test: (groupid=0, jobs=1): err= 0: pid=3567399: Sat Aug 17 09:38:55 2019
write: IOPS=1935, BW=7744KiB/s (7930kB/s)(462MiB/61143msec); 0 zone resets
slat (usec): min=9, max=700161, avg=65.17, stdev=2092.54
clat (usec): min=954, max=2578.6k, avg=16457.67, stdev=109995.03
lat (usec): min=1021, max=2578.6k, avg=16523.42, stdev=110014.91
clat percentiles (usec):
| 1.00th=[ 1254], 5.00th=[ 1434], 10.00th=[ 1549],
| 20.00th=[ 1745], 30.00th=[ 1909], 40.00th=[ 2114],
| 50.00th=[ 2376], 60.00th=[ 2704], 70.00th=[ 3228],
| 80.00th=[ 4080], 90.00th=[ 8717], 95.00th=[ 53216],
| 99.00th=[ 291505], 99.50th=[ 675283], 99.90th=[1669333],
| 99.95th=[2231370], 99.99th=[2365588]
bw ( KiB/s): min= 8, max=35968, per=100.00%, avg=9015.64,
stdev=8402.84, samples=105
iops : min= 2, max= 8992, avg=2253.90, stdev=2100.72,
samples=105
lat (usec) : 1000=0.01%
lat (msec) : 2=34.85%, 4=44.49%, 10=11.54%, 20=1.84%, 50=1.81%
lat (msec) : 100=3.27%, 250=1.13%, 500=0.42%, 750=0.19%, 1000=0.08%
cpu : usr=1.42%, sys=6.63%, ctx=123309, majf=0, minf=283
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
=64=0.0%
submit : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%,
8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
=64=0.0%
issued rwts: total=0,118371,0,0
short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=7744KiB/s (7930kB/s), 7744KiB/s-7744KiB/s
(7930kB/s-7930kB/s), io=462MiB (485MB), run=61143-61143msec
Disk stats (read/write):
rbd0: ios=0/118670, merge=0/9674, ticks=0/1894238, in_queue=1651008,
util=33.33%
On 17/8/19 8:46 am, Olivier AUDRY wrote:
Write and read with 2 hosts 4 osd :
mkfs.ext4 /dev/rbd/kube/bench
mount /dev/rbd/kube/bench /mnt/
dd if=/dev/zero of=test bs=8192k count=1000 oflag=direct
8388608000 bytes (8.4 GB, 7.8 GiB) copied, 117.541 s, 71.4 MB/s
fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randwrite
-direct=1 -runtime=60 -filename=/dev/rbd/kube/bench
WRITE: bw=45.3MiB/s (47.5MB/s), 45.3MiB/s-45.3MiB/s (47.5MB/s-
47.5MB/s), io=2718MiB (2850MB), run=60003-60003msec
fio -ioengine=libaio -name=test -bs=4k -iodepth=32 -rw=randread
-direct=1 -runtime=60 -filename=/dev/rbd/kube/bench
READ: bw=187MiB/s (197MB/s), 187MiB/s-187MiB/s (197MB/s-197MB/s),
io=10.0GiB (10.7GB), run=54636-54636msec
pgbench before : 10 transaction per second
pgbench after : 355 transaction per second
So yes it's better. SSD are INTEL SSDSC2BB48 0370.
Le samedi 17 août 2019 à 01:55 +0300, vitalif(a)yourcmc.ru a écrit :
on a new
ceph cluster with the same software and config (ansible)
on
the old hardware. 2 replica, 1 host, 4 osd.
=> New hardware : 32.6MB/s READ / 10.5MiB WRITE
=> Old hardware : 184MiB/s READ / 46.9MiB WRITE
No discussion ? I suppose I will keep the old hardware. What do you
think ? :D
In fact I don't really believe in 184 MB/s random reads with Ceph
with 4
OSDs, it's a very cool result if it's true.
Does the "new cluster on the old hardware" consist of only 1 host?
Did
you test reads before you actually wrote anything into the image so
it
was empty and reads were fast because of that?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io