Mark Lehrer <lehrer@gmail.com> writes:
One server has LSI SAS3008 [0] instead of the Perc H800, which comes with 512MB RAM + BBU. On most servers latencies are around 4-12ms (average 6ms), on the system with the LSI controller we see 20-60ms (average 30ms) latency.
Are these reads, writes, or a mixed workload? I would expect an improvement in writes, but 512MB of cache isn't likely to help much on reads with such a large data set.
It's mostly write (~20MB/s), little read (1-5 MB/s) work load. This is probably due to many people using this storage for backup.
Just as a test, you could removing the battery on one of the H800s to disable the write cache -- or else disable write caching with megaraid or equivalent.
That is certainly an interesting idea - and rereading your message and my statement above might actually explain the behaviour: - The pattern is mainly write centric, so write latency is probably the real factor - The HDD OSDs behind the raid controllers can cache / reorder writes and reduce seeks potentially So while "a raid controller" per se does probably not improve or reduce speed for ceph, "a (disk/raid) controller with a battery backed cache", might actually. In this context: is anyone here using HBAs with battery backed cache, and if yes, which controllers do you tend to use? Nico -- Sustainable and modern Infrastructures by ungleich.ch