30 Jun
2019
30 Jun
'19
10:10 p.m.
I have been experimenting using Ceph mostly for KVM image storage but I just have been
unable to get any decent RBD performance with sequential writes. Reads are mostly fine,
same with random read/write they are handled fine. I appear to be limited to around
30-40MB/s write speed on 3 hard drives (size 2). I moved my tests over to a different
single node and osd performance remained the same. I have messed with Bluestore cache, Rbd
cache and the speed level remains fairly constant apart from a burst of speed when I begin
the write test.
Looking at iostat on the host, it appears the LVM groups for the osd-block are close to
maxed out util (~ 80 % - 100 %) but the raw devices show anywhere between 10-40% and the
write speed of a single drive should be a lot higher. During my various tests I have tried
doing rados benchmarks and that gives me results way closer to what I was expecting. Also
sometimes all the IO hits a single OSD, regardless of stripe size on RBD.
After various trial and error I have tried putting the DB on an SSD and while it is not my
ideal situtation (another point of failure), it appears to of helped, slightly. So one of
my theories of it being the rockstore DB sync write was not the sole issue. However
interestingly I noticed that CephFS gives me greater performance than RBD even just using
a raw image file for a vm (random IO does suffer however), it is still not exactly what I
would of hoped for my setup, around ~ 80MB-120MB on 3 osd - size 2.
This was also the case with mimic and was one of the reasons I upgraded and remade all the
bluestore OSDs, but without any significant gains. I was previously using ZFS and able to
maintain over 200MB/s with a similar setup.
I have been trying to crunch this issue for many weeks, however I feel I have just been
going in circles and I wanted to use RBD as most of my data is VM storage. Using CephFS
for the performance boost would mean I loose many of the advantages of RBD like
snapshotting, ease of use between multiple pools, plus CephFS random write io is nowhere
near as good.
Is there any easy way to make RBD perform as well as CephFS, or to use cephfs with the
above benefits.
Ceph version
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)