Hi Joe,
I looked into perf script[0], trying to extract information related to the c2c test, in an attempt to make your "
grep ceph_test_c2c t.script |grep ldlat |sed -e 's/^.*LCK//' |awk '{s+=$2;c+=1}END {print s " " c " " s/c}'" line more flexible. The first problem I had was to figure out which fields are displayed by default:
the --fields does not say and there does not seem to be a way to show the
field labels. I took a look at the sources[1] and while the beginning (command name, pid/tid, cpu) can be guessed easily, it becomes blurry towards the end of a line like:
ceph_test_c2c 184970 [000] 617917.424796: 37057 cpu/mem-stores/P: 7ffcf821fef4 5080144 L1 hit|SNP N/A|TLB N/A|LCK N/A 0 55629c5f8708 [unknown] (/home/loic/ceph/build/bin/ceph_test_c2c) 5b4f81ef4
It looked like I could loose myself into this and did not investigate further.
It is very easy to get lost. When I shared that "perf script" output with you, I never expected that you'd be using it. The output of "perf script" is pretty cryptic and not well documented.
When I used the "perf script" command, it only worked because of your very unique test case, which had only one cacheline being contended. I have never used it before because no other program I ever looked at only had one cacheline with contention.
Can that "perf script" technique be used in a larger program with more contended cachelines? Perhaps, but it wouldn't be my preferred approach.
I did not followup on your suggestion to run perf on a running Ceph cluster either (sorry to disapoint :-( ) because I'd like to complete this simple task first. Trying to undertake anything more ambitious would be difficult.
However! As you suggested I modified the test[2] with your new shell script and ran it on a real server (128 cores, 700+GB RAM). You will find the
result in dropbox.redhat.com under the name ceph-c2c-jmario-2021-04-24-20-13.tar.gz. Hopefully they are consistent: I can't judge for myself because the content of the perf report is still quite mysterious. The metric I
set is satisfied (optimization is at least twice faster) and it is good enough to propose this for integration in the Ceph test suite. The test will fail if the optimization is accidentally disabled by a regression.
But when I look at your data, I still only see the small test case. I assumed you would be running the c2c script on a large fast server where Ceph was under a heavy real workload.
Is that something you can do?
The risk of the small test case is that it may not represent the contention situations in the larger Ceph codebase.
How about if we try this approach:
1) You run my c2c script on a large fast server running Ceph in a heavy workoad. I expect to see many cachelines being contended.
2) I analyze the results, and send back my findings.
3) We then have an interactive meeting where I can walk you through what I found and how I found it.
4) Then from step 4, you'll be able to better understand how you can create a test or tests that you can integrate into the Ceph test suite.
Does the above make sense?