On Tue, Aug 6, 2019 at 7:57 AM Janek Bevendorff <janek.bevendorff@uni-weimar.de> wrote:
4k req/s is too fast for a create workload on one MDS. That must include other operations like getattr.
That is rsync going through millions of files checking which ones need updating. Right now there are not actually any create operations, since I restarted the copy job.
Your parallel rsync job is only getting 150 creates per second? What was the previous throughput?
I wouldn't expect such extreme latency issues. Please share:
ceph config dump ceph daemon mds.X cache status
Config dump: https://pastebin.com/1jTrjzA9
Cache status:
{ "pool": { "items": 127688932, "bytes": 20401092561 } }
and the two perf dumps one second apart again please. Perf dump 1: https://pastebin.com/US3y6JEJ Perf dump 2: https://pastebin.com/Mm02puje
The cache size looks correct here.
Also, you said you removed the aggressive recall changes. I assume you didn't reset them to the defaults, right? Just the first suggested change (10k/1.0)?
Either seems to work.
I added two more MDSs to split the workload and got a steady 150 reqs/s after that. Then I noticed that I still had a max segments settings from one of my earlier attempts at fixing the cache runaway issue and after removing that, I got 250-500 reqs/s, sometimes up to 1.5k (per MDS).
Okay, so you're getting a more normal throughput for parallel creates on a single MDS.
However, to generate the dumps for you, I changed my max_mds setting back to 1 and reqs/s went down to 80. After re-adding the two active MDSs again, I am back at higher numbers, although not quite as much as before. But I think to remember that it took several minutes if not more until all MDSs received approximately equal load the last time.
Try pinning if possible in each parallel rsync job. Here are tracker tickets to resolve the issues you encountered: https://tracker.ceph.com/issues/41140 https://tracker.ceph.com/issues/41141 -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D