Hi Loïc
On Sat, Apr 24, 2021 at 2:22 PM Loïc Dachary <loic(a)dachary.org> wrote:
Hi Joe,
I looked into perf script[0], trying to extract information related to the
c2c test, in an attempt to make your "
grep ceph_test_c2c t.script |grep ldlat |sed -e 's/^.*LCK//' |awk
'{s+=$2;c+=1}END {print s " " c " " s/c}'" line more
flexible. The first
problem I had was to figure out which fields are displayed by default:
the --fields does not say and there does not seem to be a way to show the
field labels. I took a look at the sources[1] and while the beginning
(command name, pid/tid, cpu) can be guessed easily, it becomes blurry
towards the end of a line like:
ceph_test_c2c 184970 [000] 617917.424796: 37057
cpu/mem-stores/P: 7ffcf821fef4 5080144 L1 hit|SNP N/A|TLB
N/A|LCK N/A 0 55629c5f8708 [unknown]
(/home/loic/ceph/build/bin/ceph_test_c2c) 5b4f81ef4
It looked like I could loose myself into this and did not investigate
further.
It is very easy to get lost. When I shared that "perf script" output with
you, I never expected that you'd be using it. The output of "perf script"
is pretty cryptic and not well documented.
When I used the "perf script" command, it only worked because of your very
unique test case, which had only one cacheline being contended. I have
never used it before because no other program I ever looked at only had one
cacheline with contention.
Can that "perf script" technique be used in a larger program with more
contended cachelines? Perhaps, but it wouldn't be my preferred approach.
I did not followup on your suggestion to run perf on a running Ceph cluster
either (sorry to disapoint :-( ) because I'd like
to complete this simple
task first. Trying to undertake anything more ambitious would be difficult.
However! As you suggested I modified the test[2] with your new shell
script and ran it on a real server (128 cores, 700+GB RAM). You will find
the
result in
dropbox.redhat.com under the name
ceph-c2c-jmario-2021-04-24-20-13.tar.gz. Hopefully they are consistent: I
can't judge for myself because the content of the perf report is still
quite mysterious. The metric I
set is satisfied (optimization is at least twice faster) and it is good
enough to propose this for integration in the Ceph test suite. The test
will fail if the optimization is accidentally disabled by a regression.
But when I look at your data, I still only see the small test case. I
assumed you would be running the c2c script on a large fast server where
Ceph was under a heavy real workload.
Is that something you can do?
The risk of the small test case is that it may not represent the contention
situations in the larger Ceph codebase.
How about if we try this approach:
1) You run my c2c script on a large fast server running Ceph in a heavy
workoad. I expect to see many cachelines being contended.
2) I analyze the results, and send back my findings.
3) We then have an interactive meeting where I can walk you through what I
found and how I found it.
4) Then from step 4, you'll be able to better understand how you can
create a test or tests that you can integrate into the Ceph test suite.
Does the above make sense?
Joe