After reading Joe Mario's comments[0] and blog post[1] (which conveniently includes a source file to demonstrate the cacheline problems[2]), it looks like it would be useful to create a teuthology test that does something similar to the current mempool test[3] while c2c collects information. There could be two runs: * N threads looping on shard[0].bytes += value * N threads looping on shard[pick_a_shard()].bytes += value And comparing c2c reports should show the run with sharding is better. Does that sound like a reasonable approach or do you think there is a simpler way to verify the optimization works as expected ? Cheers [0] https://github.com/ceph/ceph/pull/9431 [1] https://joemario.github.io/blog/2016/09/01/c2c-blog/ [2] https://github.com/joemario/perf-c2c-usage-files/blob/master/false_sharing_e... [3] https://github.com/ceph/ceph/blob/3fb62b1a4c5d2a51db3f2b6276c7c48b3d8da201/s... On 17/03/2021 10:16, Loïc Dachary wrote:
Hi Adam,
Thanks for the pointer: very informative indeed :-) perf c2c seems to be a good tool and works on my laptop. I'll have to dig deeper to figure out what it does exactly. To be continued!
Cheers
On 17/03/2021 08:48, Adam Kupczyk wrote:
Loic,
Here is some discussion on cache false sharing that you may find valuable: https://github.com/ceph/ceph/pull/9431 <https://github.com/ceph/ceph/pull/9431> . Maybe it is possible to turn perf c2c into some systemic check?
Best regards, Adam
On Tue, 16 Mar 2021 at 22:59, Loïc Dachary <loic@dachary.org <mailto:loic@dachary.org>> wrote:
On 16/03/2021 22:01, Mark Nelson wrote: > Hi Loïc, > > > I don't have any tests specifically for this, but looking at #39057, I wonder if something could be crafted using the check_shard_select test that Adam wrote as a template. What do you think? That sounds like a good idea. And maybe using cachegrind[0] also?
[0] https://www.valgrind.org/docs/manual/cg-manual.html <https://www.valgrind.org/docs/manual/cg-manual.html> > > > Mark > > > On 3/16/21 3:52 PM, Loïc Dachary wrote: >> Hi Mark, >> >> While trying to figure out a random failure in the mempool tests[0] introduced when fixing a bug in how mempool selects shards holding the byte count of a given pool[1] earlier this year, I was intrigued by this "cache line ping pong" problem[2]. And I wonder if you have some kind of benchmark, somewhere in your toolbox, that someone could use to demonstrate the problem. Maybe such a code could be adapted to show the benefit of the optimization implemented in mempool? >> >> Cheers >> >> [0] https://tracker.ceph.com/issues/49781#note-9 <https://tracker.ceph.com/issues/49781#note-9> >> [1] https://github.com/ceph/ceph/pull/39057/files <https://github.com/ceph/ceph/pull/39057/files> >> [2] https://www.drdobbs.com/parallel/understanding-and-avoiding-memory-issues/21... <https://www.drdobbs.com/parallel/understanding-and-avoiding-memory-issues/212400410> >> >> >> _______________________________________________ >> Dev mailing list -- dev@ceph.io <mailto:dev@ceph.io> >> To unsubscribe send an email to dev-leave@ceph.io <mailto:dev-leave@ceph.io> > _______________________________________________ > Dev mailing list -- dev@ceph.io <mailto:dev@ceph.io> > To unsubscribe send an email to dev-leave@ceph.io <mailto:dev-leave@ceph.io>
-- Loïc Dachary, Artisan Logiciel Libre
_______________________________________________ Dev mailing list -- dev@ceph.io <mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io <mailto:dev-leave@ceph.io>
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Loïc Dachary, Artisan Logiciel Libre