Thanks, Anthony. Each R740 as follows: - 1 BOSS Boot Drive (Proxmox) - 1 Samsung PM1725a NVMe SSD (used as a DB Disk for each OSD) - 24 OSDs — SAS SSD 12/GBs - 1 PERC H730P RAID card (in HBA mode) - 25GB/s two port NICs (NDC, not PCIe) I was considering replacing the PERC H740P RAID card with the HBA330 Mini Monolithic; but only if it would improve performance. I have a Cisco Nexus9000 switches with capability to go to 40Gbe or 100Gbe. I was considering adding 40GBe NICs to each node in the cluster, to be used as a Private network for Ceph. Then doing some type of link aggregation with the two 25Gbe ports on the NDC NIC. I do not know much about Link Aggregation/LCP, but I'm currently studying it, as it seems to be a necessity in a production environment. Thoughts? Regards, [image] Anthony Fecarotta Founder & President [image] anthony@linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/> On Sun May 18, 2025, 08:09 PM GMT, Anthony D'Atri <mailto:anthony.datri@gmail.com> wrote:
My experience is that an IR HBA with FBWC and supercap can somewhat accelerate latency with slow media. Wrapping each drive in a VD to enable WB caching, though, is extra work and confounds drive metrics.
It’s also been that cache modules / BBUs can be flaky, and these things really need additional monitoring (hint: iDRAC isn’t enough)
If you don’t already have optional cache / BBU aka CV, I wouldn’t spend the $ retrofitting, especially on systems from 6-7 years ago. Put the $ toward faster networking or NVMe-enabled systems. The R640 / R740 can be had in all-NVMe chassis and is an inexpensive way to break out of the SAS/SATA trap.
There are no fewer than 40 R740 chassis types, if you count risers. This document
https://dl.dell.com/manuals/common/dellemc-nvme-io-topologies-poweredge.pdf <https://dl.dell.com/manuals/common/dellemc-nvme-io-topologies-poweredge.pdf>
gives you a taste.
Assuming that these are H330mini or similar, and all SAS/SATA, I personally would set the HBA mode / personality to JBOD / passthrough / HBA and act like the RoC isn’t there. Including boot drives if you don’t have BOSS. ymmv.
Whatever you do, I advise using DSU to update firmware. Old firmware on these LSI / PERC / Avago / Broadcom HBAs can present significant issues.
On May 18, 2025, at 8:11 AM, Anthony Fecarotta <anthony@linehaul.ai> wrote:
Does running a RAID controller in HBA mode (not to be confused with IT mode) impact Ceph performance compared to using a dedicated HBA card? Is there any documentation or benchmarking data showing improved performance with true HBA hardware?
For what it's worth my cluster is on Dell PowerEdge R740 machines.
Thank you for your insights.
Regards, [image] Anthony Fecarotta Founder & President [image] anthony@linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io