hello

I got a huge difference of read/write performance on my ceph cluster when I use rbd. 

rbd bench reach the limit of my cluster (1gbps network) when the performance in mapped rbd is very low. 30MB/s.

ceph version : ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)

kernel version : 4.19.0-5-amd64

ceph -s

  cluster:

    id:     fdf6eba0-cde9-401c-8901-751eb1e8b7d2

    health: HEALTH_OK

  services:

    mon: 3 daemons, quorum gra3-k8s001,sbg2-k8s001,rbx4-k8s001 (age 41h)

    mgr: sbg2-k8s001(active, since 46h), standbys: gra3-k8s001, rbx4-k8s001

    osd: 10 osds: 10 up (since 41h), 10 in (since 41h)

    rgw: 3 daemons active (gra3-k8s001, rbx4-k8s001, sbg2-k8s001)

  data:

    pools:   8 pools, 240 pgs

    objects: 109.14k objects, 229 GiB

    usage:   694 GiB used, 3.0 TiB / 3.7 TiB avail

    pgs:     240 active+clean

here the bench

rbd create --size=10G kube/bench

rbd feature disable kube/bench object-map fast-diff deep-flatten

rbd -p kube bench kube/bench --io-type write --io-size 8192 --io-threads 256 --io-total 10G --io-pattern seq

elapsed:   110  ops:  1310720  ops/sec: 11884.39  bytes/sec: 97356926.64

this reach the 1gbps limit of my network. Looks good.

let's test random write

rbd -p kube bench kube/bench --io-type write --io-size 8192 --io-threads 256 --io-total 10G --io-pattern rand

elapsed:   125  ops:  1310720  ops/sec: 10416.31  bytes/sec: 85330446.58

this reach the 1gbps limit of my network. Looks good.

rbd -p kube bench kube/bench --io-type read --io-size 8192 --io-threads 256 --io-total 10G --io-pattern seq

elapsed:    40  ops:  1310720  ops/sec: 32712.03  bytes/sec: 267976974.35

this is great perf !

let's try rand

rbd -p kube bench kube/bench --io-type read --io-size 8192 --io-threads 256 --io-total 10G --io-pattern rand

elapsed:    29  ops:  1310720  ops/sec: 43859.78  bytes/sec: 359299337.92

this is great perf !

But when I mount the image :

rbd map bench  --pool kube --name client.admin

/sbin/mkfs.ext4  /dev/rbd/kube/bench 

 mount /dev/rbd/kube/bench /mnt/

cd /mnt/

dd if=/dev/zero of=test bs=8192k count=100 oflag=direct

838860800 bytes (839 MB, 800 MiB) copied, 24.6185 s, 34.1 MB/s

34.1MB/s vs 85MB/s ....