- The pattern is mainly write centric, so write latency is probably the real factor - The HDD OSDs behind the raid controllers can cache / reorder writes and reduce seeks potentially
OK that makes sense. Unfortunately, re-ordering HDD writes without a battery backup is kind of dangerous -- writes need to happen in order or the filesystem will punish you when you least expect it. This is the whole point of the battery backup - to make sure that out-of-order writes get written to disk even if there is a power loss in the middle of writing the controller-write-cache data in an HDD-optimized order. Your use case is ideal for an SSD-based WAL -- though it may be difficult to beat the cost of H800s these days.
In this context: is anyone here using HBAs with battery backed cache, and if yes, which controllers do you tend to use?
I almost always use MegaRAID-based controllers (such as the H800). Good luck, Mark On Tue, Apr 20, 2021 at 2:28 PM Nico Schottelius <nico.schottelius@ungleich.ch> wrote:
Mark Lehrer <lehrer@gmail.com> writes:
One server has LSI SAS3008 [0] instead of the Perc H800, which comes with 512MB RAM + BBU. On most servers latencies are around 4-12ms (average 6ms), on the system with the LSI controller we see 20-60ms (average 30ms) latency.
Are these reads, writes, or a mixed workload? I would expect an improvement in writes, but 512MB of cache isn't likely to help much on reads with such a large data set.
It's mostly write (~20MB/s), little read (1-5 MB/s) work load. This is probably due to many people using this storage for backup.
Just as a test, you could removing the battery on one of the H800s to disable the write cache -- or else disable write caching with megaraid or equivalent.
That is certainly an interesting idea - and rereading your message and my statement above might actually explain the behaviour:
- The pattern is mainly write centric, so write latency is probably the real factor - The HDD OSDs behind the raid controllers can cache / reorder writes and reduce seeks potentially
So while "a raid controller" per se does probably not improve or reduce speed for ceph, "a (disk/raid) controller with a battery backed cache", might actually.
In this context: is anyone here using HBAs with battery backed cache, and if yes, which controllers do you tend to use?
Nico
-- Sustainable and modern Infrastructures by ungleich.ch