However, now my client processes are basically in
constant I/O wait
state and the CephFS is slow for everybody. After I restarted the copy
job, I got around 4k reqs/s and then it went down to 100 reqs/s with
everybody waiting their turn. So yes, it does seem to help, but it
increases latency by a magnitude.
Addition: I reduced the number to 256K and the cache size started
inflating instantly (with about 140 reqs/s). So I reset it to 512K and
the cache size started reducing slowly, though with fewer reqs/s.
So I guess it is solving the problem, but only by trading it off against
severe latency issues (order of magnitude as we saw).