Anthony, Thank you very much for your comments; they were very helpful. It made me reconsider some aspects of the configuration, and it also helped me see that I wasn't too far off in general. I'll respond to some of your suggestions, explaining my reasons.

Indeed, I know from experience that LFF spinners don't cut it for boot drives. Even with strawberries.

My experience with LFF spinners is the same; when I set up the first cluster, it was the only economically viable option.

Do you strictly need a second cluster? Or could you just constrain your pools on the existing cluster based on deviceclass?

I want to set up a second cluster since the first one is on leased hardware, and I want to be prepared for when it expires.

SMCI offers chassis that are NVMe-only I think. The above I think comes with an HBA you don't need or want.

The HBA is only for the operating system disks; the rest of the NVMe U.2 drives are connected to the PCIe bus.

The Kingstons are cost-effective, but last I looked up the specs they were kinda meh. Beats spinners though. This is more CPU and more RAM than you need for 10xNVMe unless you're also going to run RGW or other compute on them.

I know there are better drives, but these U.2 drives are more affordable, just like the server. I did an exercise with U.3 drives that had double the capacity, and each server cost twice as much. It's a good option, but with my current budget, it's not feasible.

...

Two Intel NIC E810-XXVDA2 25GbE Dual Port (2 x SFP28) PCIe 4.0 x8 cards

Why two?

...

Connected to 2 MikroTik CRS518-16XS-2XQ-RM switches at 100GbE per server Connection to OpenStack would be via 4 x 10GB to our core switch.

Might 25GE be an alternative?

Again, for economic reasons, I installed 2 NIC 25GB Dual Port to create a LAG and achieve a 100GB connection. The connection to the core switch is also through another LAG with 4 x 10GB (and if needed, I can add more ports). This is because our core switch doesn't have any free SFP ports. For now, I can only purchase Mikrotik switches due to their cost, but in the future, when the leasing period ends, I'll consider other types of switches. Thank you so much Gustavo ________________________________ De: Anthony D'Atri <anthony.datri@gmail.com> Enviado: viernes, 6 de octubre de 2023 16:52 Para: Gustavo Fahnle <gfahnle@nubi2go.com> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Asunto: Re: [ceph-users] Hardware recommendations for a Ceph cluster

Currently, I have an OpenStack installation with a Ceph cluster consisting of 4 servers for OSD, each with 16TB SATA HDDs. My intention is to add a second, independent Ceph cluster to provide faster disks for OpenStack VMs.

Indeed, I know from experience that LFF spinners don't cut it for boot drives. Even with strawberries.

The idea for this second cluster is to exclusively provide RBD services to OpenStack

Do you strictly need a second cluster? Or could you just constrain your pools on the existing cluster based on deviceclass?

For the OSDs, I'm thinking of starting with 3 or 4 servers, specifically Supermicro AS-1114S-WN10RT,

SMCI offers chassis that are NVMe-only I think. The above I think comes with an HBA you don't need or want.

each with:

1 AMD EPYC 7713P Gen 3 processor (64 Core, 128 Threads, 2.0GHz) 256GB of RAM 2 x NVME 1TB for the operating system 10 x NVME Kingston DC1500M U.2 7.68TB for the OSDs

The Kingstons are cost-effective, but last I looked up the specs they were kinda meh. Beats spinners though. This is more CPU and more RAM than you need for 10xNVMe unless you're also going to run RGW or other compute on them.

Two Intel NIC E810-XXVDA2 25GbE Dual Port (2 x SFP28) PCIe 4.0 x8 cards

Why two?

Connected to 2 MikroTik CRS518-16XS-2XQ-RM switches at 100GbE per server Connection to OpenStack would be via 4 x 10GB to our core switch.

Might 25GE be an alternative?

I would like to hear opinions about this configuration, recommendations, criticisms, etc.

If any of you have references or experience with any of the components in this initial configuration, they would be very welcome.

Thank you very much in advance.

Gustavo Fahnle

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

AFAIK the standing recommendation for all flash setups is to prefer fewer but faster cores

Hrm, I think this might depend on what you’re solving for. This is the conventional wisdom for MDS for sure. My sense is that OSDs can use multiple cores fairly well, so I might look at the cores * GHz product. Especially since this use-case sounds like long-tail performance probably isn’t worth thousands. Only four OSD servers, Neutron, Kingston. I don’t think the OP has stated any performance goals other than being more suitable to OpenStack instances than LFF spinners.

so something like a 75F3 might be yielding better latency. Plus you probably want to experiment with partitioning the NVMEs and running multiple OSDs per drive - either 2 or 4.

Mark Nelson has authored a series of blog posts that explore this in great detail over a number of releases. TL;DR: with Quincy or Reef, especially, my sense is that multiple OSDs per NVMe device is not the clear win that it once was, and just eats more RAM. Mark has also authored detailed posts about OSD performance vs cores per OSD, though IIRC those are for one OSD in isolation. In a real-world cluster, especially one this small, I suspect that replication and the network will be bottlenecks before either of the factors discussed above. ymmv.

On Sat, 7 Oct 2023 at 08:23, Gustavo Fahnle <gfahnle@nubi2go.com> wrote:

...

Hi,

Currently, I have an OpenStack installation with a Ceph cluster consisting of 4 servers for OSD, each with 16TB SATA HDDs. My intention is to add a second, independent Ceph cluster to provide faster disks for OpenStack VMs. The idea for this second cluster is to exclusively provide RBD services to OpenStack. I plan to start with a cluster composed of 3 mon/mgr nodes similar to what we currently have (3 virtualized servers with VMware) with 4 cores, 8GB of memory, 80GB disk and 10GB network each server. In the current cluster, these nodes have low resource consumption, less than 10% CPU usage, 40% memory usage, and less than 100Mb/s of network usage.

For the OSDs, I'm thinking of starting with 3 or 4 servers, specifically Supermicro AS-1114S-WN10RT, each with:

1 AMD EPYC 7713P Gen 3 processor (64 Core, 128 Threads, 2.0GHz) 256GB of RAM 2 x NVME 1TB for the operating system 10 x NVME Kingston DC1500M U.2 7.68TB for the OSDs Two Intel NIC E810-XXVDA2 25GbE Dual Port (2 x SFP28) PCIe 4.0 x8 cards Connected to 2 MikroTik CRS518-16XS-2XQ-RM switches at 100GbE per server Connection to OpenStack would be via 4 x 10GB to our core switch.

I would like to hear opinions about this configuration, recommendations, criticisms, etc.

If any of you have references or experience with any of the components in this initial configuration, they would be very welcome.

Thank you very much in advance.

Gustavo Fahnle

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io