One server has LSI SAS3008 [0] instead of the Perc H800, which comes with 512MB RAM + BBU. On most servers latencies are around 4-12ms (average 6ms), on the system with the LSI controller we see 20-60ms (average 30ms) latency.
Are these reads, writes, or a mixed workload? I would expect an improvement in writes, but 512MB of cache isn't likely to help much on reads with such a large data set. Just as a test, you could removing the battery on one of the H800s to disable the write cache -- or else disable write caching with megaraid or equivalent. On Mon, Apr 19, 2021 at 12:21 PM Nico Schottelius <nico.schottelius@ungleich.ch> wrote:
Good evening,
I've to tackle an old, probably recurring topic: HBAs vs. Raid controllers. While generally speaking many people in the ceph field recommend to go with HBAs, it seems in our infrastructure the only server we phased in with an HBA vs. raid controller is actually doing worse in terms of latency.
For the background: we have many Perc H800+MD1200 [1] systems running with 10TB HDDs (raid0, read ahead, writeback cache). One server has LSI SAS3008 [0] instead of the Perc H800, which comes with 512MB RAM + BBU. On most servers latencies are around 4-12ms (average 6ms), on the system with the LSI controller we see 20-60ms (average 30ms) latency.
Now, my question is, are we doing some inherently wrong with the SAS3008 or does in fact the cache help to possible reduce seek time?
We were considering to move more towards LSI HBAs to reduce maintenance effort, however if we have a factor of 5 in latency between the two different systems, it might be better to stay on the H800 path for disks.
Any input/experiences appreciated.
Best regards,
Nico
[0] 05:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) Subsystem: Dell 12Gbps HBA Kernel driver in use: mpt3sas Kernel modules: mpt3sas
[1] 08:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05) Subsystem: Dell PERC H800 Adapter Kernel driver in use: megaraid_sas Kernel modules: megaraid_sas
-- Sustainable and modern Infrastructures by ungleich.ch _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io