Hi, I'm a bit confused by your question the 'drive bays' or backplane is the same for an NVMe system, it's either a SATA/SAS/NVME backplane or a NVMe backplane. I don't understand why you believe that my configuration has to be 3.5" as it isn't. It's a 16x2.5" chassis with two H755N controllers (one for each set of 8 drives). The H755N controller is a hardware raid adapter for NVMe. I hope this clarifies the confusion. Thank you, -Drew -----Original Message----- From: Frank Schilder <frans@dtu.dk> Sent: Thursday, July 11, 2024 9:57 AM To: Drew Weaver <drew.weaver@thenap.com>; 'ceph-users@ceph.io' <ceph-users@ceph.io> Subject: Re: Repurposing some Dell R750s for Ceph Hi Drew, as far as I know Dell's drive bays for RAID controllers are not the same as the drive bays for CPU attached disks. In particular, I don't think they have that config for 3.5" drive bays and your description sounds a lot like that's what you have. Are you trying to go from 16x2.5" HDD to something like 24xNVMe? Maybe you could provide a bit more information here, like (links to) the wiring diagrams you mentioned? From the description I cannot entirely deduce what exactly you have and where you want to go to. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Drew Weaver <drew.weaver@thenap.com> Sent: Thursday, July 11, 2024 3:16 PM To: 'ceph-users@ceph.io' Subject: [ceph-users] Repurposing some Dell R750s for Ceph Hello, We would like to repurpose some Dell PowerEdge R750s for a Ceph cluster. Currently the servers have one H755N RAID controller for each 8 drives. (2 total) I have been asking their technical support what needs to happen in order for us to just rip out those raid controllers and cable the backplane directly to the motherboard/PCIe lanes and they haven't been super enthusiastic about helping me. I get it just buy another 50 servers, right? No big deal. I have the diagrams that show how each configuration should be connected, I think I just need the right cable(s), my question is has anyone done this work before and was it worth it? Also bonus if anyone has an R750 that has the drives directly connected to the backplane and can find the part number of the cable that connects the backplane to the motherboard I would greatly appreciate that part number. My sales guys are "having a hard time locating it". Thanks, -Drew _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi, Isn’t the supported/recommended configuration to use an HBA if you have to but never use a RAID controller? The backplane is already NVMe as the drives installed in the system currently are already NVMe. Also I was looking through some diagrams of the R750 and it appears that if you order them with the RAID controller(s) the bandwidth between the backplane and the system is hamstrung to some degree because of the cables they are using so even if I could configure them in NON raid it would still be suboptimal. Thanks for the information. -Drew From: John Jasen <jjasen@gmail.com> Sent: Thursday, July 11, 2024 10:06 AM To: Drew Weaver <drew.weaver@thenap.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Repurposing some Dell R750s for Ceph retrofitting the guts of a Dell PE R7xx server is not straightforward. You could be looking into replacing the motherboard, the backplane, and so forth. You can probably convert the H755N card to present the drives to the OS, so you can use them for Ceph. This may be AHCI mode, pass-through mode, non-RAID device, or some other magic words in the raid configuration utility. This should be in the raid documentation, somewhere. On Thu, Jul 11, 2024 at 9:17 AM Drew Weaver <drew.weaver@thenap.com<mailto:drew.weaver@thenap.com>> wrote: Hello, We would like to repurpose some Dell PowerEdge R750s for a Ceph cluster. Currently the servers have one H755N RAID controller for each 8 drives. (2 total) I have been asking their technical support what needs to happen in order for us to just rip out those raid controllers and cable the backplane directly to the motherboard/PCIe lanes and they haven't been super enthusiastic about helping me. I get it just buy another 50 servers, right? No big deal. I have the diagrams that show how each configuration should be connected, I think I just need the right cable(s), my question is has anyone done this work before and was it worth it? Also bonus if anyone has an R750 that has the drives directly connected to the backplane and can find the part number of the cable that connects the backplane to the motherboard I would greatly appreciate that part number. My sales guys are "having a hard time locating it". Thanks, -Drew _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>

Isn’t the supported/recommended configuration to use an HBA if you have to but never use a RAID controller?

That may be something I added to the docs. My contempt for RAID HBAs knows no bounds ;) Ceph doesn’t care. Passthrough should work fine, I’ve done that for tends of thousands of OSDs, albeit on different LSI HBA SKUs.

Okay it seems like we don't really have a definitive answer on whether it's OK to use a RAID controller or not and in what capacity.

It’s okay to use it if that’s what you have. For new systems, eschew the things. They cost money for something you can do with MD for free and are finicky. Look back on the list a few years for my litany of why I hate the things, all based on personal experience ;)

Passthrough meaning:

Are you saying that it's OK to use a raid controller where the disks are in non-RAID mode?

That’s the best approach in your situation, where you have servers already.

Are you saying that it's OK to use a raid controller where each disk is in its own RAID-0 volume?

It’s OK to do that, but there are drawbacks and you should do passthrough instead if you can: * Operational hassles creating those wrappers and tearing them down when deploying OSDs and replacing drives * Firmware updates may be confounded * iostat’s already-marginal stats are confounded * Hassles getting SMART-style stats

I'm just trying to clarify a little bit. You can imagine that nobody wants to be that user that does this against the documentation's guidelines and then something goes terribly wrong.

Dell’s documentation tells you how to set passthrough. If you’ve seen docs that tell you not to, please do send a link as I’d like to see them. Dell qualified a certain unnamed now-EOL SSD, yet I underwent a long and arduous engagement with the manufacturer to fix a firmware design flaw. So I don’t place much stock in Dell recommendations. They REEEEAALLLY likes to stuff RoC HBAs down our throats, and they mark them up outrageously. — aad

Thanks again, -Drew

-----Original Message----- From: Anthony D'Atri <aad@dreamsnake.net> Sent: Thursday, July 11, 2024 7:24 PM To: Drew Weaver <drew.weaver@thenap.com> Cc: John Jasen <jjasen@gmail.com>; ceph-users@ceph.io Subject: Re: [ceph-users] Repurposing some Dell R750s for Ceph

...

Isn’t the supported/recommended configuration to use an HBA if you have to but never use a RAID controller?

That may be something I added to the docs. My contempt for RAID HBAs knows no bounds ;)

Ceph doesn’t care. Passthrough should work fine, I’ve done that for tends of thousands of OSDs, albeit on different LSI HBA SKUs.

There’s more to it than bottlenecking. RAS, man. RAS.

On Jul 12, 2024, at 3:58 PM, John Jasen <jjasen@gmail.com> wrote:

How large of a ceph cluster are you planning on building, and what network cards/speeds will you be using?

A lot of the talk about RAID HBA pass-through being sub-optimal probably won't be your bottleneck unless you're aiming for a large cluster at 100Gb/s speeds, in my opinion.

On Fri, Jul 12, 2024 at 12:02 PM Drew Weaver <drew.weaver@thenap.com <mailto:drew.weaver@thenap.com>> wrote:

...

Okay it seems like we don't really have a definitive answer on whether it's OK to use a RAID controller or not and in what capacity.

Passthrough meaning:

Are you saying that it's OK to use a raid controller where the disks are in non-RAID mode? Are you saying that it's OK to use a raid controller where each disk is in its own RAID-0 volume?

I'm just trying to clarify a little bit. You can imagine that nobody wants to be that user that does this against the documentation's guidelines and then something goes terribly wrong.

Thanks again, -Drew

-----Original Message----- From: Anthony D'Atri <aad@dreamsnake.net <mailto:aad@dreamsnake.net>> Sent: Thursday, July 11, 2024 7:24 PM To: Drew Weaver <drew.weaver@thenap.com <mailto:drew.weaver@thenap.com>> Cc: John Jasen <jjasen@gmail.com <mailto:jjasen@gmail.com>>; ceph-users@ceph.io <mailto:ceph-users@ceph.io> Subject: Re: [ceph-users] Repurposing some Dell R750s for Ceph

...

Isn’t the supported/recommended configuration to use an HBA if you have to but never use a RAID controller?

That may be something I added to the docs. My contempt for RAID HBAs knows no bounds ;)

Ceph doesn’t care. Passthrough should work fine, I’ve done that for tends of thousands of OSDs, albeit on different LSI HBA SKUs.

Agree with everything Robin wrote here. RAID HBAs FTL. Even in passthrough mode, it’s still an [absurdly expensive] point of failure, but a server in the rack is worth two on backorder. Moreover, I’m told that it is possible to retrofit with cables and possibly an AIC mux / expander. e.g. https://www.ebay.com/itm/176400760681 Granted, I haven’t done this personally so I can’t speak to the BOM and procedure. For OSD nodes it probably isn’t worth the effort. Some of the LSI^H^H^H^HPERC HBAs — to my astonishment — don’t have a passthrough setting/mode. This document though implies that this SKU does. https://www.dell.com/support/manuals/en-ae/poweredge-r7525/perc11_ug/technical-specifications-of-perc-11-cards?guid=guid-aaaf8b59-903f-49c1-8832-f3997d125edf&lang=en-us You should be able to set individual drives to passthrough: storcli64 /call /eall /sall set jbod=on or depending on the SKU and storcli revision, for the whole HBA storcli64 /call set personality=JBOD racadm set Storage.Controller.1.RequestedControllerMode HBA or racadm set Storage.Controller.1.RequestedControllerMode EnhancedHBA then jobqueue create RAID.Integrated.1-1 server action power cycle LSI and Dell have not been particularly consistent with these beasts. — aad

...

Hello,

We would like to repurpose some Dell PowerEdge R750s for a Ceph cluster.

Currently the servers have one H755N RAID controller for each 8 drives. (2 total) The N variant of H755N specifically? So you have 16 NVME drives in each server?

...

I have been asking their technical support what needs to happen in order for us to just rip out those raid controllers and cable the backplane directly to the motherboard/PCIe lanes and they haven't been super enthusiastic about helping me. I get it just buy another 50 servers, right? No big deal. I don't think the motherboard has enough PCIe lanes to natively connect all the drives: the RAID controller effectively functioned as a expander, so you needed less PCIe lanes on the motherboard.

As the quickest way forward: look for passthrough / single-disk / RAID0 options, in that order, in the controller management tools (perccli etc).

I haven't used the N variant at all, and since it's NVME presented as SCSI/SAS, I don't want to trust the solution of reflashing the controller for IT (passthrough) mode.

-- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation President & Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io