You can structure your crush map so that you get multiple EC chunks per host in a way that you can still survive a host outage outage even though you have fewer hosts than k+1 For example if you run an EC=4+2 profile on 3 hosts you can structure your crushmap so that you have 2 chunks per host. This means even if one host is down you are still guaranteed to have 4 chunks available. If you then set min_size = 4 you can still operate your cluster in that situation - albeit risky since any additional failure in that time will lead to data loss. However in a highly constrained setup it might be a trade-off that's worth it for you. There have been examples of this on this mailing list in the past. On Wed, 6 Dec 2023 at 12:11, Rich Freeman <r-ceph@rich0.org> wrote:
On Tue, Dec 5, 2023 at 6:35 AM Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> wrote:
Ok, so I've misunderstood the meaning of failure domain. If there is no way to request using 2 osd/node and node as failure domain, with 5 nodes k=3+m=1 is not secure enough and I will have to use k=2+m=2, so like a raid1 setup. A little bit better than replication in the point of view of global storage capacity.
I'm not sure what you mean by requesting 2osd/node. If the failure domain is set to the host, then by default k/m refer to hosts, and the PGs will be spread across all OSDs on all hosts, but with any particular PG only being present on one OSD on each host. You can get fancy with device classes and crush rules and such and be more specific with how they're allocated, but that would be the typical behavior.
Since k/m refer to hosts, then k+m must be less than or equal to the number of hosts or you'll have a degraded pool because there won't be enough hosts to allocate them all. It won't ever stack them across multiple OSDs on the same host with that configuration.
k=2,m=2 with min=3 would require at least 4 hosts (k+m), and would allow you to operate degraded with a single host down, and the PGs would become inactive but would still be recoverable with two hosts down. While strictly speaking only 4 hosts are required, you'd do better to have more than that since then the cluster can immediately recover from a loss, assuming you have sufficient space. As you say it is no more space-efficient than RAID1 or size=2, and it suffers write amplification for modifications, but it does allow recovery after the loss of up to two hosts, and you can operate degraded with one host down which allows for somewhat high availability.
-- Rich _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io