Hi Loïc, I don't have any tests specifically for this, but looking at #39057, I wonder if something could be crafted using the check_shard_select test that Adam wrote as a template.  What do you think? Mark On 3/16/21 3:52 PM, Loïc Dachary wrote:

Loic, Here is some discussion on cache false sharing that you may find valuable: https://github.com/ceph/ceph/pull/9431 . Maybe it is possible to turn perf c2c into some systemic check? Best regards, Adam On Tue, 16 Mar 2021 at 22:59, Loïc Dachary <loic(a)dachary.org> wrote:

After reading Joe Mario's comments[0] and blog post[1] (which conveniently includes a source file to demonstrate the cacheline problems[2]), it looks like it would be useful to create a teuthology test that does something similar to the current mempool test[3] while c2c collects information. There could be two runs: * N threads looping on shard[0].bytes += value * N threads looping on shard[pick_a_shard()].bytes += value And comparing c2c reports should show the run with sharding is better. Does that sound like a reasonable approach or do you think there is a simpler way to verify the optimization works as expected ? Cheers [0] https://github.com/ceph/ceph/pull/9431 [1] https://joemario.github.io/blog/2016/09/01/c2c-blog/ [2] https://github.com/joemario/perf-c2c-usage-files/blob/master/false_sharing_… [3] https://github.com/ceph/ceph/blob/3fb62b1a4c5d2a51db3f2b6276c7c48b3d8da201/… On 17/03/2021 10:16, Loïc Dachary wrote: -- Loïc Dachary, Artisan Logiciel Libre

Hi Loïc, It looks like the commands that I posted in my last reply got wrapped and garbled. Try grabbing them from this script instead. http://people.redhat.com/jmario/scratch/run_c2c_ceph.sh Joe

Hi Loïc, That sounds great. Thank you. Joe

I could not wait and started working on it right away. An issue was opened to track progress[0] and not spam the mailing list. To be continued. [0] https://tracker.ceph.com/issues/49896 On 18/03/2021 19:00, Loïc Dachary wrote: -- Loïc Dachary, Artisan Logiciel Libre

Hi Joe, Thanks for the comment, it makes perfect sense. I'm eager to work on it but I'll have be patient and wait until next week. To be continued! On 25/03/2021 13:40, jmario(a)redhat.com wrote: -- Loïc Dachary, Artisan Logiciel Libre

Hi Joe, The test program[0] was run with the commands you provided[1] and a 30 seconds "warm up" period. The output was uploaded to dropbox.redhat.com in a file named ceph-c2c-jmario-2021-04-04-16-28.tar.gz. There are two directories: * with-sharding is with the optimization turned on * without-sharding is without the optimization I took a quick look and saw a difference in the perf_c2c_a_all_user_phys_data.out files. Without the optimization there is only one line and with the optimization there are 8 (the number of threads). The interpretation of this difference is beyond me and I'm very curious to read what you make of it, as well as the rest of the output :-) A machine is dedicated to this test and doing nothing else. Now that it's ready I'll be able to adjust whatever you suggest the same day. Thanks again for your patience and willingness to share your expertise! Cheers [0] https://lab.fedeproxy.eu/ceph/ceph/-/blob/wip-mempool-cacheline-49781/src/t… [1] https://lab.fedeproxy.eu/ceph/ceph/-/blob/wip-mempool-cacheline-49781/qa/st… ================================================ # #        ----------- Cacheline ----------      Tot  ------- Load Hitm -------    Total    Total    Total  ---- Stores ----  ----- Core Load Hit -----  - LLC Load Hit --  - RMT Load Hit --  --- Load Dram ---- # Index             Address  Node  PA cnt     Hitm    Total  LclHitm  RmtHitm  records    Loads   Stores    L1Hit   L1Miss       FB       L1       L2    LclHit  LclHitm    RmtHit  RmtHitm       Lcl       Rmt # .....  ..................  ....  ......  .......  .......  .......  .......  .......  .......  .......  .......  .......  .......  .......  .......  ........  .......  ........  .......  ........  ........ #       0      0x7fffff04c200     0       1  100.00%    35593    35593        0   140432    60712    79720    79720        0        0    25114        0         5    35593         0        0         0         0 =================================================       Shared Cache Line Distribution Pareto ================================================= # #        ----- HITM -----  -- Store Refs --  --------- Data address ---------                      ---------- cycles ----------    Total       cpu                                         Shared #   Num  RmtHitm  LclHitm   L1 Hit  L1 Miss              Offset  Node  PA cnt        Code address  rmt hitm  lcl hitm      load  records       cnt                          Symbol         Object        Source:Line  Node # .....  .......  .......  .......  .......  ..................  ....  ......  ..................  ........  ........  ........  .......  ........  ..............................  .............  .................  .... #   -------------------------------------------------------------       0        0    35593    79720        0      0x7fffff04c200   -------------------------------------------------------------            0.00%  100.00%  100.00%    0.00%                0x30     0       1      0x55669fda67ba         0       550       450   140432         4  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0 =================================================            Shared Data Cache Line Table ================================================= # #        ----------- Cacheline ----------      Tot  ------- Load Hitm -------    Total    Total    Total  ---- Stores ----  ----- Core Load Hit -----  - LLC Load Hit --  - RMT Load Hit --  --- Load Dram ---- # Index             Address  Node  PA cnt     Hitm    Total  LclHitm  RmtHitm  records    Loads   Stores    L1Hit   L1Miss       FB       L1       L2    LclHit  LclHitm    RmtHit  RmtHitm       Lcl       Rmt # .....  ..................  ....  ......  .......  .......  .......  .......  .......  .......  .......  .......  .......  .......  .......  .......  ........  .......  ........  .......  ........  ........ #       0      0x7ffecdcb4700     0   71586  100.00%    22548    22548        0    84925    28141    56784    56784        0        0     5592        0         1    22548         0        0         0         0 =================================================       Shared Cache Line Distribution Pareto ================================================= # #        ----- HITM -----  -- Store Refs --  --------- Data address ---------                      ---------- cycles ----------    Total       cpu                                         Shared #   Num  RmtHitm  LclHitm   L1 Hit  L1 Miss              Offset  Node  PA cnt        Code address  rmt hitm  lcl hitm      load  records       cnt                          Symbol         Object        Source:Line  Node # .....  .......  .......  .......  .......  ..................  ....  ......  ..................  ........  ........  ........  .......  ........  ..............................  .............  .................  .... #   -------------------------------------------------------------       0        0    22548    56784        0      0x7ffecdcb4700   -------------------------------------------------------------            0.00%   27.38%   15.11%    0.00%                 0x0     0       1      0x55cc50aa67ba         0       274       289    16378         2  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0            0.00%    0.00%   14.67%    0.00%                 0x4     0       1      0x55cc50aa67ba         0         0         0     8331         1  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0            0.00%   26.40%   15.26%    0.00%                 0x8     0       1      0x55cc50aa67ba         0       279       289    16572         1  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0            0.00%   13.32%   14.72%    0.00%                 0xc     0       1      0x55cc50aa67ba         0       548       406    11974         2  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0            0.00%   10.36%   12.93%    0.00%                0x10     0       1      0x55cc50aa67ba         0       423       299    10135         1  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0            0.00%    9.82%   12.42%    0.00%                0x14     0       1      0x55cc50aa67ba         0       407       288     9638         1  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0            0.00%   12.72%   14.89%    0.00%                0x18     0       1      0x55cc50aa67ba         0       546       410    11897         1  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0 On 18/03/2021 18:25, Loïc Dachary wrote: -- Loïc Dachary, Artisan Logiciel Libre

Hi Loïc: I just took a quick look, was surprised to see no contention, and then realized that this was done on an AMD server. AMD never added their support for the low level perf events that are similar to Intel's PEBS events. The AMD cpus do have those similar features, and we've talked with them about getting them added to the perf tool, but so far we've not seen it. I'm sorry if I didn't stress the "Intel" piece strongly enough. Can you find an Intel server to rerun this test? Any Intel server in the last 10 years (Ivy Bridge or newer) should work. However for this testing, the newer and faster the server is, the more pronounced the contention will be. So if you can get on a Cascade Lake or Skylake server, that would be best. Joe

Hi Loic: A little more clarification, as I dug in deeper. I was wrong on the cpu type. It is an Intel cpu. But led me astray was the uname and the /proc/cmdline: uname: Linux enough 5.10.0-5-amd64 #1 SMP Debian 5.10.24-1 (2021-03-19) x86_64 GNU/Linux /proc/cmdline: BOOT_IMAGE=/vmlinuz-5.10.0-5-amd64 root=/dev/mapper/enough--vg-root ro quiet But looking at the lscpu, I do see it's an Intel cpu. (Sorry for not looking at that first before sent my earlier reply.) It appears the Debian amd64 kernel does not support the perf events needed for this. Is it possible for you to run this on a recent CentOS-8 OS version? I also see the system you used is a small, desktop level system with only 4 cpus (2 cores). If you can get a larger server, you should see more contention, which would help with this effort. Thanks, Joe

On 04/04/2021 17:46, jmario(a)redhat.com wrote: I'm relieved. Is there a simple way to verify that the kernel has support for perf events ? I'd like to try Ubuntu because it's been a while since I installed a CentOS. Yes, once the tests are ready I'll  run them on an Intel server with 128 CPU :-) -- Loïc Dachary, Artisan Logiciel Libre

Hi Loïc: Looking further, there is something in those files. It's just one small cacheline, but there is something there. I guess I'm not used to seeing c2c being run on a laptop, nor am I used to seeing so few samples, or even so little in the kernel. And I apologize for my quick initial mistaken analysis. Here's what it looks like is happening. Correct me if I'm wrong. In your "without-sharding" version of Ceph, you had 8 threads in the ceph_test_c2c binary all contending for the same lock located at offset 0 in a cacheline. And then, in the "with-sharding" version of Ceph, you changed it so that each thread would act on its own copy of the 4-byte lock. Unfortunately, the "with-sharding" version likely didn't help, because all those 8 locks are packed into the same cacheline, with each lock being 4 bytes away from the last one. If that is true, then you need to rewrite the code such that all the locks are located by themselves in their own cacheline. Is the above assumption correct? Also, I'm going to update the "run_c2c_ceph.sh" script I gave you. The "-g" flag isn't working as it should (known issue). Joe On Sun, Apr 4, 2021 at 12:21 PM Loïc Dachary &lt;loic(a)dachary.org&gt; wrote:

On 04/04/2021 20:24, Joe Mario wrote: small cacheline, but there is something there.  I guess I'm not used to seeing c2c being run on a laptop, nor am I used to seeing so few samples, or even so little in the kernel.  Yes, absolutely right. I changed the variable to be 128 bytes aligned[0], is it ok? Maybe there is a constant somewhere that provides this number (number of bytes to be "cache aligned") so it is not hard coded? The output is uploaded in ceph-c2c-jmario-2021-04-04-22-13.tar.gz and hopefully looks better. [0] https://lab.fedeproxy.eu/ceph/ceph/-/commit/54f3a4d0ece0e817bf9308617040f8b… P.S. no need to apologize: I'm grateful for the quick feedback on a Sunday :-) The "-g" flag isn't working as it should (known issue). -- Loïc Dachary, Artisan Logiciel Libre

Morning Joe, On 05/04/2021 02:15, Joe Mario wrote: to 128 bytes (e.g. 2 cachelines instead of 1). database vendors pad out to 128 bytes because of this as well. Thanks for explaining: it makes sense now. It did not run, indeed. The version with sharding is faster and it finished before the measures started. The first observable evidence of the optimization, exciting :-) I changed the test program so that it keeps running forever and will be killed by the caller when it is no longer needed. The output was uploaded in ceph-c2c-jmario-2021-04-05-09-26.tar.gz Cheers -- Loïc Dachary, Artisan Logiciel Libre

And the commit with the benchmark test: https://lab.fedeproxy.eu/ceph/ceph/-/commit/b8ab6380adfc028da8166704dbc1755… On 05/04/2021 10:48, Loïc Dachary wrote: -- Loïc Dachary, Artisan Logiciel Libre

[snip] And if there are N CPUs and N*2 threads, the only way to avoid cacheline contention is if all of these threads use a different (cacheline aligned) variable. Is that correct? If it is correct, I wonder if cacheline contention has a linear impact on performances. If there is cacheline contention on 1 variable with N threads on N CPUS, will the performance degradation be the same if there is cacheline contention on 10 variables and the same number of threads & CPUS? Or will it get worse because there is some sort of amplification? That's because perf was instructed to ignore any load instructions that took faster than 70 machine cycles to complete.  The reasoning is at those low levels of machine cycles, there is no contention, so why burden the perf tool execution and data collection with the extra processing of fast loads that aren't relevant to finding cacheline contention.   I mention this for completeness.  The drop from 751 to 84 machine cycles is significant. Thanks for the crystal clear explanation. I'd like for the test script to extract those number from the files produced by perf c2c. How do you suggest I go about this? These 128 bytes are not used, they are just padding to make sure nothing else is stored there. Cheers [0] https://lab.fedeproxy.eu/ceph/ceph/-/blob/wip-mempool-cacheline-49781/src/i…

Hi Nathan, The repos built for https://shaman.ceph.com/builds/ceph/wip-mempool-cacheline-49781/ a few weeks ago expired, would you be so kind as to push the latest from my repo to the ceph-ci repository[1]? Thanks again for your help! [0] https://lab.fedeproxy.eu/ceph/ceph/-/blob/wip-mempool-cacheline-49781/ [1] https://github.com/ceph/ceph-ci/tree/wip-mempool-cacheline-49781 On 04/04/2021 16:49, Loïc Dachary wrote: -- Loïc Dachary, Artisan Logiciel Libre

Hi Loïc On Sat, Apr 24, 2021 at 2:22 PM Loïc Dachary &lt;loic(a)dachary.org&gt; wrote: It is very easy to get lost. When I shared that "perf script" output with you, I never expected that you'd be using it. The output of "perf script" is pretty cryptic and not well documented. When I used the "perf script" command, it only worked because of your very unique test case, which had only one cacheline being contended. I have never used it before because no other program I ever looked at only had one cacheline with contention. Can that "perf script" technique be used in a larger program with more contended cachelines? Perhaps, but it wouldn't be my preferred approach. I did not followup on your suggestion to run perf on a running Ceph cluster But when I look at your data, I still only see the small test case. I assumed you would be running the c2c script on a large fast server where Ceph was under a heavy real workload. Is that something you can do? The risk of the small test case is that it may not represent the contention situations in the larger Ceph codebase. How about if we try this approach: 1) You run my c2c script on a large fast server running Ceph in a heavy workoad. I expect to see many cachelines being contended. 2) I analyze the results, and send back my findings. 3) We then have an interactive meeting where I can walk you through what I found and how I found it. 4) Then from step 4, you'll be able to better understand how you can create a test or tests that you can integrate into the Ceph test suite. Does the above make sense? Joe

On Sat, Apr 24, 2021 at 11:13:39PM +0200, Loïc Dachary wrote: Sure, here you go: https://github.com/ceph/ceph/pull/41014

Hi Nathan, Josh noticed that one line could be removed[0] from the test script. I did it and repushed[1]. Would you be so kind as to push the change to GitHub? Thanks for your help! [0] https://github.com/ceph/ceph/pull/41014#pullrequestreview-645521549 [1] https://lab.fedeproxy.eu/ceph/ceph/-/tree/wip-mempool-cacheline-49781 On 25/04/2021 17:10, Loïc Dachary wrote: -- Loïc Dachary, Artisan Logiciel Libre

Hi Kefu, On 29/04/2021 18:14, kefu chai wrote: 143, in main in prepare_and_schedule in schedule_suite in collect_jobs in __init__ in __init__ in _init_from_config it’s own faces for choosing a random distro, and my test happened to pick rhel 8.3 for testing, but the distro name was overridden by the one specified by c2c.yaml. That’s why I had a combination of Ubuntu 8.3. I just took the liberty to push another commit to the pull request in hope to test sooner. If it looks sane to you, could you include it in your commit? Or I can do this with your permission. This is perfect! I had doubts about running this against something other than Ubuntu and was not sure which package to include. You have my permission (and gratitude) to squash the commits together. Cheers -- Loïc Dachary, Artisan Logiciel Libre

Hi Loïc: Great to see this is moving forward. A few questions and comments: 1) I do see the test code measuring the runtime differences between the sharding and non-sharding cases. Were you going to add test code to analyze the output of the perf c2c runs (for hottest cachelines and long load latencies). That can be challenging to figure out how to do it effectively, given all the variables involved with different test environments. 2) Is there a check to verify the test is being run on an Intel based system? 3) I see the comment in the mempool.h file: // Align shard to a cacheline. // // It would be possible to retrieve the value at runtime (for instance // with getconf LEVEL1_DCACHE_LINESIZE or grep -m1 cache_alignment // /proc/cpuinfo). It is easier to hard code the largest cache // linesize for all known processors (128 bytes). If the actual cache // linesize is smaller on a given processor, it will just waste a few // bytes. // The above is slightly incorrect about wasting a few bytes. On Intel platforms where the cacheline size is 64 bytes, it's smart to pad your hot locks out to 128 bytes. (I briefly mentioned this earlier.) The reason is because of the hardware prefetchers. When the prefetchers fetch a cacheline, they anticipate your program will also need the next cacheline, so they gratuitously fetch that one as well. They load 128 bytes instead of the requested 64 bytes. If that next cacheine is in a critical code path, any users of that line will have their cacheline copy marked invalid because the prefetchers for the earlier cacheline just overwrote it. The performance problems we see from this are rare, but are real and are difficult to diagnose. The Linux kernel and some big database vendors pad all their hot locks and variables out to 128 bytes just for this reason. 4) I see the test code retained all the commands I put into the c2c.sh file so that I could understand what the system environment was like. You only need to keep them and the kernel perf c2c commands if you feel you need them. Nice to see this work is happening. Thanks, Joe On Fri, Apr 30, 2021 at 10:04 AM Loïc Dachary &lt;loic(a)dachary.org&gt; wrote: