14 Aug
2019
14 Aug
'19
7:37 p.m.
hello
I got a huge difference of read/write performance on my ceph cluster
when I use rbd.
rbd bench reach the limit of my cluster (1gbps network) when the
performance in mapped rbd is very low. 30MB/s.
ceph version : ceph version 14.2.2
(4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)
kernel version : 4.19.0-5-amd64
ceph -s
cluster:
id: fdf6eba0-cde9-401c-8901-751eb1e8b7d2
health: HEALTH_OK
services:
mon: 3 daemons, quorum gra3-k8s001,sbg2-k8s001,rbx4-k8s001 (age
41h)
mgr: sbg2-k8s001(active, since 46h), standbys: gra3-k8s001, rbx4-
k8s001
osd: 10 osds: 10 up (since 41h), 10 in (since 41h)
rgw: 3 daemons active (gra3-k8s001, rbx4-k8s001, sbg2-k8s001)
data:
pools: 8 pools, 240 pgs
objects: 109.14k objects, 229 GiB
usage: 694 GiB used, 3.0 TiB / 3.7 TiB avail
pgs: 240 active+clean
here the bench
rbd create --size=10G kube/bench
rbd feature disable kube/bench object-map fast-diff deep-flatten
rbd -p kube bench kube/bench --io-type write --io-size 8192 --io-
threads 256 --io-total 10G --io-pattern seq
elapsed: 110 ops: 1310720 ops/sec: 11884.39 bytes/sec:
97356926.64
this reach the 1gbps limit of my network. Looks good.
let's test random write
rbd -p kube bench kube/bench --io-type write --io-size 8192 --io-
threads 256 --io-total 10G --io-pattern rand
elapsed: 125 ops: 1310720 ops/sec: 10416.31 bytes/sec:
85330446.58
this reach the 1gbps limit of my network. Looks good.
rbd -p kube bench kube/bench --io-type read --io-size 8192 --io-threads
256 --io-total 10G --io-pattern seq
elapsed: 40 ops: 1310720 ops/sec: 32712.03 bytes/sec:
267976974.35
this is great perf !
let's try rand
rbd -p kube bench kube/bench --io-type read --io-size 8192 --io-threads
256 --io-total 10G --io-pattern rand
elapsed: 29 ops: 1310720 ops/sec: 43859.78 bytes/sec:
359299337.92
this is great perf !
But when I mount the image :
rbd map bench --pool kube --name client.admin
/sbin/mkfs.ext4 /dev/rbd/kube/bench
mount /dev/rbd/kube/bench /mnt/
cd /mnt/
dd if=/dev/zero of=test bs=8192k count=100 oflag=direct
838860800 bytes (839 MB, 800 MiB) copied, 24.6185 s, 34.1 MB/s
34.1MB/s vs 85MB/s ....