Thank you for the detailed response. At my primary facility I am already running a five node ceph cluster (among other server backup servers and GPU stations), so it seems I am already past the point of benefiting from such a design. This cluster is currently connected to a Cisco N9K-C92160YC-X, using the 25gbe interfaces, with plenty to spare.I think the lack of throughput is a fundamental flaw with the way I have the OSDs and PGs configured. Regards, * Anthony Fecarotta* Founder & President [image: phone-icon] anthony@linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/> On Tue, Feb 4, 2025 at 12:55 AM Janne Johansson <icepic.dz@gmail.com> wrote:
I read a Ceph Benchmark paper (
https://www.proxmox.com/en/downloads/proxmox-virtual-environment/documentati... )
where they demonstrated, among other things, the performance of using a Full-Mesh Network Schema for Ceph on a three node cluster.
Is this method used at an enterprise level? I am seriously considering it.
The idea that you can connect three nodes by each having a dual-port card and have them talk to "all the other nodes" without having a router or a switch is a neat corner case for having 3 hosts (ie, with 2 it is obvious that you can reach the other, and with 4 or more OSD hosts you are really soon going to get a switch or a router) so while this setup works, it also limits you a lot in terms of how easy it will be to expand it later.
If you ask ceph admins, they will often tell you to never go below repl=3 (or use EC K+M where M is 1) because of the inherent risks of it and also to have at least one OSD node more than your repl factor or the sum of K+M for EC pools so that the cluster can recover by itself when a node or disk dies.
Those two pieces of advice makes this setup less attractive, since it scales badly for more than 3 OSD hosts, while at the same time, if you care about your data, you will not run repl=2 (at least not for long).
I get that you can have a really decent quick-to-setup proxmox storage in this way and the results were really nice but if you put this in place on your company and the setup becomes popular, will you scale by building yet another next to it without them being able to see eachother, or do you want to scale out (like the rest of us ceph admins do) to tens or hundreds of OSD hosts?
Do note that the gains from scaling out your ceph cluster was visible at the end of this paper, they added some old benchmark hosts and got lots better results as expected, at least for reads.
This is what the nice part about adding OSD hosts to ceph means, you get more cpu for the computing parts of crypto or checksumming, you get more ram for caches, you add to the total network bandwidth and you get better/faster/easier recovery when a single OSD host dies, apart from the simple fact of "I also got more free disk space".
So that design works, and as long as you don't have faults or surprises, it will perform nicely. It's just that as years pass by, the chances of storage systems not seeing surprises or faults diminish very rapidly. Drives die, PSUs fail, networks split. Stuff happens, and the more popular this setup becomes internally at your enterprise, the less forgiving will the clients be when a simple PSU faults causes the whole storage to be degraded until someone can get a replacement PSU just because the cluster had no place to recover into as you run it with "just as much as needed and nothing else" when you build with 3 hosts.
Compare this to some 16 drive raid box. If you really want to be able to sleep when a drive dies you would be running raid-10 or so, but also have hot spares, and quite possibly cold spares lying around too.
In the ceph case, the repl=3 equals the raid-1 (but with 3 copies) and the striping would be to have several OSD hosts, but when you have repl=3 and 3 hosts it equals not having any hot spares in that raid-box. When a drive dies, you can limp along by getting data off the other disks, but you are at risk the whole time.
If your raid box has hot spares, it will automagically start repairing into this, just as a 4-OSD-host ceph cluster with repl=3 pools will recover data into the remaining 3 hosts. The difference is that the ceph cluster will use the 4th host too, but will have the ability to run on 3 (given enough free space) and can get back into a fully redundant and performant state by itself so you can sleep over night while it recovers and then wait for replacement parts while the cluster is not in a panic mode.
-- May the most significant bit of your life be positive.