I have been experimenting using Ceph mostly for KVM image storage but I just have been unable to get any decent RBD performance with sequential writes. Reads are mostly fine, same with random read/write they are handled fine. I appear to be limited to around 30-40MB/s write speed on 3 hard drives (size 2). I moved my tests over to a different single node and osd performance remained the same. I have messed with Bluestore cache, Rbd cache and the speed level remains fairly constant apart from a burst of speed when I begin the write test.

Looking at iostat on the host, it appears the LVM groups for the osd-block are close to maxed out util (~ 80 % - 100 %) but the raw devices show anywhere between 10-40% and the write speed of a single drive should be a lot higher. During my various tests I have tried doing rados benchmarks and that gives me results way closer to what I was expecting. Also sometimes all the IO hits a single OSD, regardless of stripe size on RBD.

After various trial and error I have tried putting the DB on an SSD and while it is not my ideal situtation (another point of failure), it appears to of helped, slightly. So one of my theories of it being the rockstore DB sync write was not the sole issue. However interestingly I noticed that CephFS gives me greater performance than RBD even just using a raw image file for a vm (random IO does suffer however), it is still not exactly what I would of hoped for my setup, around ~ 80MB-120MB on 3 osd - size 2.

This was also the case with mimic and was one of the reasons I upgraded and remade all the bluestore OSDs, but without any significant gains. I was previously using ZFS and able to maintain over 200MB/s with a similar setup.

I have been trying to crunch this issue for many weeks, however I feel I have just been going in circles and I wanted to use RBD as most of my data is VM storage. Using CephFS for the performance boost would mean I loose many of the advantages of RBD like snapshotting, ease of use between multiple pools, plus CephFS random write io is nowhere near as good.