[...]
Isn't a bigger k+m
The biggerr the more "correlating" IO operations across a number of OSDs.
a good things for performance
As to speed higher 'k' is good for large IO, especially sequential reads, up to a point (latency dispersion), but it is usually very bad for writes smaller than 'k' blocks. Later you say "block" as in VM disk/container images, so if they are read-only that *may* be fine, but if they are not read-only then usually stateful VMs and containers do a lot of smaller writes. Then time to recovery and overall latency depend as "Janne Johanson" wrote as to what fraction of your failure domains are involved in recovery after a failure and the higher the worse. The classic 3-way replication works well for that and the next best is k=2,m=2.
if network and hardware not the bottleneck ?
That is a very lucky situation, most Ceph clusters reported here usually are rather underprovisioned, especially as to IOPS (more rarely as to networking, sometimes as to RAM per OSD server). It would be useful to know what the OSDs are and how many you have on each of the 9 servers.
Is there any particular reason why the failure domain is host but pods with 3 hosts get shut down?
I feel that it will give the same result as pods failure domain with the gain of loosing 3 hosts
That is a decision that can go either way but not quite arbitrarily. The questions are whether online availability is more important than data durability and whether single host failures are more likely than whose pod failures. That sort of determines whether to choose 3 or 9 or N (N = OSDs) failure domains.
Perhaps 'k=2,m=2' or for the very brave 'k=4,m=2'. But again it all depends on expected data and workload profiles.
Just downsize the profile to 4+3 as this cluster is in test phase
That 'm=3' seems a bit excessive to me and can be quite expensive. Also 4+3 is still 7-wide out of 9 failure domains and will still have wide consequences if one full pod goes down. If you really really want 'm=3' perhaps 'k=3' would be a better match.
(I want to keep this failure host domain and I believe this is the best to do ?)
Odds are that is good but then you should not be surprised or dismayed with the consequences of taking down a full pod.