we expanded the cluster to 12 nodes with 28 osds
2 or 3 OSDs per node? Are these like Raspberry Pis, or maybe converged (Proxmox?)
but we are still running on EC 3+2, since we started cluster with 6 nodes.
There are ways to fake failure domains (hosts, racks, etc) when you *know* that a cluster will be expanding, albeit with increased risk until it does.
We would like to change it to EC 6+3 or even EC 8+3? Is this reasonable or will 8+3 take too much CPU/network capacities?
That depends on several things. * Are you looking to change the profile to get more usable capacity? * Are you looking to change the profile to get more durability / sustain more failures? * What is your hardware like? CPU, drive size/model, network links * What kind of data are you storing? RBD volumes? CephFS? Big files? Tiny files? * Which Ceph release are you running?
Should I use isa/isa-l instead of jerasure?
For new pools I would suggest that. The two are very similar, but ISA is maintained while jerasure is abandonware.
Can anyone share experience with 8+3?
It goes to 11 ;) That said, props for having k+m < the number of failure domains. This gives you better RAS, and lets the size of those failure domains vary (somewhat) without stranding capacity. Having k+m=failure domains is a very common ... well I won't go so far as to call it a mistake, but let's say suboptimal strategy. I helped someone with 10 dense HDD nodes running 8+2. Adding one more (and later more) node made a big difference. If you are storing mostly very small files/ RGW objects, 8+3 will "waste" a high percentage of underlying storage. https://markhpc.github.io/2019/11/21/BlueFS-Space-Amplification.html If there are a few small files mixed in with much bigger ones, that becomes less of a concern. This space amplification drove the reduction of the default min_alloc_size from 64KB to 4KB in the O/P era. If you are running Tentacle and the pool has the ec optimization flag turned on, this space amp will be much reduced since RADOS will no longer add null padding. A wide EC profile will make scrubs and recovery slower due to the gridlock effect -- scrubs and recovery rely on reservations, and there will be more collision and thus queueing. A wide EC profile will result in slower writes. Your media and workload are factors in whether or not that's a problem for you. Whether your data is on CephFS, RGW, and/or RBD informs the actual process for moving. Please share more detail. There has been talk of being able to do this transparently with Umbrella, but it's hearsay until it's merged.
Kind regards, Rok _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io