we expanded the cluster to 12 nodes with 28 osds
2 or 3 OSDs per node? Are these like Raspberry Pis, or maybe converged (Proxmox?)
28 osds per node.
Oh, gotcha. The way you wrote sounded MUCH smaller. So you have 336 OSDs.
but we are still running on EC 3+2, since we started cluster with 6 nodes.
There are ways to fake failure domains (hosts, racks, etc) when you *know* that a cluster will be expanding, albeit with increased risk until it does.
We use host failure domain for now.
We would like to change it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too much CPU/network capacities?
That depends on several things.
* Are you looking to change the profile to get more usable capacity? * Are you looking to change the profile to get more durability / sustain more failures? Not necessarily, we would rather have a responsive rgw for example during failures, so when there is backfill/recovery going on that rgw does not stop responding.
The wider profile may actually make that worse. Since you mention RGW and only mention HDDs, does that mean that your index pool is on HDDs? There's a good chance that was a factor. HDDs have very limited IOPS and all of the RGW pools that aren't bucket data really should be on SSDs. I speculate that client ops, backfill, and RGW ops were DoSing each other. An expedient way to add those would be something like https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcTRl12SIsjlOJTId1V6Zk... A PCI-e AIC that accepts 2x NVMe M.2 drives, assuming that your motherboard can do PCIe bifurcation. You could populate each with like 2x 1.9T Micron 7450 and place the RGW pools on those. For this purpose having more OSDs is more beneficial than having larger OSDs. You could easily see the size of your existing pools to help gauge what you need.
It happened a few times. e.g. slow/faulty disk from on which reweight was just set
You aren't using "ceph osd reweight" or "ceph osd reweight-by-utilization" are you? Those are legacies and should be avoided. You want the REWEIGHT value on each OSD to be 1.0000 and to use the manager balancer. ceph config set mgr mgr/balancer/upmap_max_deviation 1
, was the reason that rgw stopped responding, since some queue filled up ... because ceph was unable to read from faulty pgs or smth...
Probably saturation, one of the false economies of HDDs.
* What is your hardware like? CPU, drive size/model, network links
There is a mix of: - Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 20Gb network bonds
Check your bonds' xmit_hash_policy. This often defaults to layer2, but often should be 2+3 or 3+4. Run iptraf-ng, downarrow to General interface statistics, and see if TX traffic is roughly balanced.
- AMD EPYC 9254 24-Core Processor with 50Gb network bonds
with mostly old 6T Seagate ST6000NM0095 drives, but we are just in the process rotating them.
* What kind of data are you storing? RBD volumes? CephFS? Big files? Tiny files?
RGW, S3 users, backup data, nextcloud storage or just storage for placing files. Mostly tiny files, however backups with >100GB or even TB files can be found also.
Does that mean you're only using RGW? No RBD, no CephFS? This informs the process by which you would migrate data between pools. If you don't have capacity concerns, maybe migration isn't worth the trouble.
usage: 775 TiB used, 1.0 PiB / 1.8 PiB avail
* Which Ceph release are you running?
Reef 18.2.8
Rok