That configuration is optimal for maximizing recovery times and write latency: since each EC set will span 8 hosts of 9, every time a host becames unavailable virtually all PGs will become degraded, and even with all host available any write will involve nearly all hosts.
This kind of optimal configuration is popular with those who think that the original design assumption of Ceph of having many, many more hosts and OSDs than the replication factor or stripe width was wasteful and cowardly. :-)
Isn't a bigger k+m a good things for performance if network and hardware not the bottleneck ?
Is there any particular reason why the failure domain is host but pods with 3 hosts get shut down?
I feel that it will give the same result as pods failure domain with the gain of loosing 3 hosts
Depends for what. But it is difficult to say without knowing the expected data and workload profiles.
Will be mostly bloc (virtual machine and kubernetes pods)
I might opine that in most cases it is as unimportant as having 'min_size=2' with 'size=3'. :-) "Safely" and 'k=5' with 'min_size=5' is an interesting association :-).
Asking just in case but already knew the answer I guess
Perhaps 'k=2,m=2' or for the very brave 'k=4,m=2'. But again it all depends on expected data and workload profiles.
Just downsize the profile to 4+3 as this cluster is in test phase (I want to keep this failure host domain and I believe this is the best to do ?) ________________________________ De : Peter Grandi via ceph-users <ceph-users@ceph.io> Envoyé : lundi 27 juillet 2026 12:15:47 À : list Linux fs Ceph Objet : [ceph-users] Re: Ec pool min size
We have a 9 hosts (3 per pod) cluster with 1 pool in EC 5+3 (host domain failure),
That configuration is optimal for maximizing recovery times and write latency: since each EC set will span 8 hosts of 9, every time a host becames unavailable virtually all PGs will become degraded, and even with all host available any write will involve nearly all hosts. This kind of optimal configuration is popular with those who think that the original design assumption of Ceph of having many, many more hosts and OSDs than the replication factor or stripe width was wasteful and cowardly. :-)
I expected that shutting down 1 pod
Is there any particular reason why the failure domain is host but pods with 3 hosts get shut down?
wouldn't affect our pg but I just discovered that the EC pool is in min_size 6 while trying it. Is min_size 6 important in this case ?
Depends for what. But it is difficult to say without knowing the expected data and workload profiles. I might opine that in most cases it is as unimportant as having 'min_size=2' with 'size=3'. :-)
Could I safely lower it to 5
"Safely" and 'k=5' with 'min_size=5' is an interesting association :-).
or using another EC k+m would suit better?
Perhaps 'k=2,m=2' or for the very brave 'k=4,m=2'. But again it all depends on expected data and workload profiles. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[...]
Isn't a bigger k+m
The biggerr the more "correlating" IO operations across a number of OSDs.
a good things for performance
As to speed higher 'k' is good for large IO, especially sequential reads, up to a point (latency dispersion), but it is usually very bad for writes smaller than 'k' blocks. Later you say "block" as in VM disk/container images, so if they are read-only that *may* be fine, but if they are not read-only then usually stateful VMs and containers do a lot of smaller writes. Then time to recovery and overall latency depend as "Janne Johanson" wrote as to what fraction of your failure domains are involved in recovery after a failure and the higher the worse. The classic 3-way replication works well for that and the next best is k=2,m=2.
if network and hardware not the bottleneck ?
That is a very lucky situation, most Ceph clusters reported here usually are rather underprovisioned, especially as to IOPS (more rarely as to networking, sometimes as to RAM per OSD server). It would be useful to know what the OSDs are and how many you have on each of the 9 servers.
Is there any particular reason why the failure domain is host but pods with 3 hosts get shut down?
I feel that it will give the same result as pods failure domain with the gain of loosing 3 hosts
That is a decision that can go either way but not quite arbitrarily. The questions are whether online availability is more important than data durability and whether single host failures are more likely than whose pod failures. That sort of determines whether to choose 3 or 9 or N (N = OSDs) failure domains.
Perhaps 'k=2,m=2' or for the very brave 'k=4,m=2'. But again it all depends on expected data and workload profiles.
Just downsize the profile to 4+3 as this cluster is in test phase
That 'm=3' seems a bit excessive to me and can be quite expensive. Also 4+3 is still 7-wide out of 9 failure domains and will still have wide consequences if one full pod goes down. If you really really want 'm=3' perhaps 'k=3' would be a better match.
(I want to keep this failure host domain and I believe this is the best to do ?)
Odds are that is good but then you should not be surprised or dismayed with the consequences of taking down a full pod.
participants (2)
-
GLE, Vivien
-
pg＠ceph.list.sabi.co.UK