On 16/03/2021 22:01, Mark Nelson wrote: That sounds like a good idea. And maybe using cachegrind[0] also? [0] https://www.valgrind.org/docs/manual/cg-manual.html -- Loïc Dachary, Artisan Logiciel Libre

Hi Adam, Thanks for the pointer: very informative indeed :-) perf c2c seems to be a good tool and works on my laptop. I'll have to dig deeper to figure out what it does exactly. To be continued! Cheers On 17/03/2021 08:48, Adam Kupczyk wrote: -- Loïc Dachary, Artisan Logiciel Libre

Hi Joe, I can't tell you how happy I am that you're around to help understand this :-) In the spirit of making baby steps to better understand what's going on, I'd like to run a small part of Ceph[0] that is designed to be optimized and avoid cacheline ping pong. The code running this part would be in a test (similar to an existing one[1]). It would be launched on a single Intel machine (as part of a teuthology run, the integration test tool specific to Ceph) and use the commands you suggest. After the first run I'll send the data to you for interpretation (sounds like consulting an oracle :-) ). With your help I'm hoping the integration test will assert that running the mempool with the optimization is at least X% faster than without the optimization and fail otherwise. That would be very helpful to guard against accidental regressions. Once this first goal is achieved, collecting data from a Ceph cluster running under load could follow the same methodology. I'm sure Mark Nelson will be most interested in this more ambitious target. If I'm not mistaken, the commands (let say they are in a ceph-c2c.sh script) should be run like this: * Run the software (be it mempool simulation or Ceph under load), let it warm up * Run ceph-c2c.sh (it won't take more than a minute or so to complete) * Collect the data and save them * Kill the software Is that correct? Cheers [0] https://github.com/ceph/ceph/blob/2b21735498c98299d5ce383011c3dbe25aaee70f/… [1] https://github.com/ceph/ceph/blob/2b21735498c98299d5ce383011c3dbe25aaee70f/… On 18/03/2021 16:40, jmario(a)redhat.com wrote: -- Loïc Dachary, Artisan Logiciel Libre

Ok then: I'll start working on it tomorrow and ping you when the data is ready. It may take a few weeks because of other obligations but it will happen :-) On 18/03/2021 18:42, jmario(a)redhat.com wrote: -- Loïc Dachary, Artisan Logiciel Libre

Hi Loïc: One quick comment. Normally a script with many commands isn't needed to detect cacheline contention. The reason I gave you one is because I know nothing about your environment, the topology, the system, or the load you're running. The many commands in that script should give me enough information such that I don't come back a 2nd time asking you to rerun something because I needed more information. All it takes is one simple "perf c2c record ..." command to examine cacheline contention. And my goal is to help you see it yourself. Let me know when you have something. Thank you. Joe

Hi Nathan, I drafted a c2c standalone teuthology test[0]. Would you be so kind as to push it to GitHub on my behalf so that it builds the corresponding packages? I'm sorry to bother you with this but I don't know if there is another way to build the packages and tell teuthology to get them. Cheers [0] https://lab.fedeproxy.eu/ceph/ceph/-/tree/wip-mempool-cacheline-49781 On 25/03/2021 14:04, Loïc Dachary wrote: -- Loïc Dachary, Artisan Logiciel Libre

Hi Nathan, The repos built for https://shaman.ceph.com/builds/ceph/wip-mempool-cacheline-49781/ a few weeks ago expired, would you be so kind as to push the latest from my repo to the ceph-ci repository[1]? Thanks again for your help! [0] https://lab.fedeproxy.eu/ceph/ceph/-/blob/wip-mempool-cacheline-49781/ [1] https://github.com/ceph/ceph-ci/tree/wip-mempool-cacheline-49781 On 04/04/2021 16:49, Loïc Dachary wrote: -- Loïc Dachary, Artisan Logiciel Libre

On Sat, Apr 24, 2021 at 11:13:39PM +0200, Loïc Dachary wrote: Sure, here you go: https://github.com/ceph/ceph/pull/41014

Hi Nathan, Josh noticed that one line could be removed[0] from the test script. I did it and repushed[1]. Would you be so kind as to push the change to GitHub? Thanks for your help! [0] https://github.com/ceph/ceph/pull/41014#pullrequestreview-645521549 [1] https://lab.fedeproxy.eu/ceph/ceph/-/tree/wip-mempool-cacheline-49781 On 25/04/2021 17:10, Loïc Dachary wrote: -- Loïc Dachary, Artisan Logiciel Libre

Hi Kefu, On 29/04/2021 18:14, kefu chai wrote: 143, in main in prepare_and_schedule in schedule_suite in collect_jobs in __init__ in __init__ in _init_from_config it’s own faces for choosing a random distro, and my test happened to pick rhel 8.3 for testing, but the distro name was overridden by the one specified by c2c.yaml. That’s why I had a combination of Ubuntu 8.3. I just took the liberty to push another commit to the pull request in hope to test sooner. If it looks sane to you, could you include it in your commit? Or I can do this with your permission. This is perfect! I had doubts about running this against something other than Ubuntu and was not sure which package to include. You have my permission (and gratitude) to squash the commits together. Cheers -- Loïc Dachary, Artisan Logiciel Libre

Hi Loïc: Great to see this is moving forward. A few questions and comments: 1) I do see the test code measuring the runtime differences between the sharding and non-sharding cases. Were you going to add test code to analyze the output of the perf c2c runs (for hottest cachelines and long load latencies). That can be challenging to figure out how to do it effectively, given all the variables involved with different test environments. 2) Is there a check to verify the test is being run on an Intel based system? 3) I see the comment in the mempool.h file: // Align shard to a cacheline. // // It would be possible to retrieve the value at runtime (for instance // with getconf LEVEL1_DCACHE_LINESIZE or grep -m1 cache_alignment // /proc/cpuinfo). It is easier to hard code the largest cache // linesize for all known processors (128 bytes). If the actual cache // linesize is smaller on a given processor, it will just waste a few // bytes. // The above is slightly incorrect about wasting a few bytes. On Intel platforms where the cacheline size is 64 bytes, it's smart to pad your hot locks out to 128 bytes. (I briefly mentioned this earlier.) The reason is because of the hardware prefetchers. When the prefetchers fetch a cacheline, they anticipate your program will also need the next cacheline, so they gratuitously fetch that one as well. They load 128 bytes instead of the requested 64 bytes. If that next cacheine is in a critical code path, any users of that line will have their cacheline copy marked invalid because the prefetchers for the earlier cacheline just overwrote it. The performance problems we see from this are rare, but are real and are difficult to diagnose. The Linux kernel and some big database vendors pad all their hot locks and variables out to 128 bytes just for this reason. 4) I see the test code retained all the commands I put into the c2c.sh file so that I could understand what the system environment was like. You only need to keep them and the kernel perf c2c commands if you feel you need them. Nice to see this work is happening. Thanks, Joe On Fri, Apr 30, 2021 at 10:04 AM Loïc Dachary <loic(a)dachary.org> wrote:

Hi Joe, The test program[0] was run with the commands you provided[1] and a 30 seconds "warm up" period. The output was uploaded to dropbox.redhat.com in a file named ceph-c2c-jmario-2021-04-04-16-28.tar.gz. There are two directories: * with-sharding is with the optimization turned on * without-sharding is without the optimization I took a quick look and saw a difference in the perf_c2c_a_all_user_phys_data.out files. Without the optimization there is only one line and with the optimization there are 8 (the number of threads). The interpretation of this difference is beyond me and I'm very curious to read what you make of it, as well as the rest of the output :-) A machine is dedicated to this test and doing nothing else. Now that it's ready I'll be able to adjust whatever you suggest the same day. Thanks again for your patience and willingness to share your expertise! Cheers [0] https://lab.fedeproxy.eu/ceph/ceph/-/blob/wip-mempool-cacheline-49781/src/t… [1] https://lab.fedeproxy.eu/ceph/ceph/-/blob/wip-mempool-cacheline-49781/qa/st… ================================================ # #        ----------- Cacheline ----------      Tot  ------- Load Hitm -------    Total    Total    Total  ---- Stores ----  ----- Core Load Hit -----  - LLC Load Hit --  - RMT Load Hit --  --- Load Dram ---- # Index             Address  Node  PA cnt     Hitm    Total  LclHitm  RmtHitm  records    Loads   Stores    L1Hit   L1Miss       FB       L1       L2    LclHit  LclHitm    RmtHit  RmtHitm       Lcl       Rmt # .....  ..................  ....  ......  .......  .......  .......  .......  .......  .......  .......  .......  .......  .......  .......  .......  ........  .......  ........  .......  ........  ........ #       0      0x7fffff04c200     0       1  100.00%    35593    35593        0   140432    60712    79720    79720        0        0    25114        0         5    35593         0        0         0         0 =================================================       Shared Cache Line Distribution Pareto ================================================= # #        ----- HITM -----  -- Store Refs --  --------- Data address ---------                      ---------- cycles ----------    Total       cpu                                         Shared #   Num  RmtHitm  LclHitm   L1 Hit  L1 Miss              Offset  Node  PA cnt        Code address  rmt hitm  lcl hitm      load  records       cnt                          Symbol         Object        Source:Line  Node # .....  .......  .......  .......  .......  ..................  ....  ......  ..................  ........  ........  ........  .......  ........  ..............................  .............  .................  .... #   -------------------------------------------------------------       0        0    35593    79720        0      0x7fffff04c200   -------------------------------------------------------------            0.00%  100.00%  100.00%    0.00%                0x30     0       1      0x55669fda67ba         0       550       450   140432         4  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0 =================================================            Shared Data Cache Line Table ================================================= # #        ----------- Cacheline ----------      Tot  ------- Load Hitm -------    Total    Total    Total  ---- Stores ----  ----- Core Load Hit -----  - LLC Load Hit --  - RMT Load Hit --  --- Load Dram ---- # Index             Address  Node  PA cnt     Hitm    Total  LclHitm  RmtHitm  records    Loads   Stores    L1Hit   L1Miss       FB       L1       L2    LclHit  LclHitm    RmtHit  RmtHitm       Lcl       Rmt # .....  ..................  ....  ......  .......  .......  .......  .......  .......  .......  .......  .......  .......  .......  .......  .......  ........  .......  ........  .......  ........  ........ #       0      0x7ffecdcb4700     0   71586  100.00%    22548    22548        0    84925    28141    56784    56784        0        0     5592        0         1    22548         0        0         0         0 =================================================       Shared Cache Line Distribution Pareto ================================================= # #        ----- HITM -----  -- Store Refs --  --------- Data address ---------                      ---------- cycles ----------    Total       cpu                                         Shared #   Num  RmtHitm  LclHitm   L1 Hit  L1 Miss              Offset  Node  PA cnt        Code address  rmt hitm  lcl hitm      load  records       cnt                          Symbol         Object        Source:Line  Node # .....  .......  .......  .......  .......  ..................  ....  ......  ..................  ........  ........  ........  .......  ........  ..............................  .............  .................  .... #   -------------------------------------------------------------       0        0    22548    56784        0      0x7ffecdcb4700   -------------------------------------------------------------            0.00%   27.38%   15.11%    0.00%                 0x0     0       1      0x55cc50aa67ba         0       274       289    16378         2  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0            0.00%    0.00%   14.67%    0.00%                 0x4     0       1      0x55cc50aa67ba         0         0         0     8331         1  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0            0.00%   26.40%   15.26%    0.00%                 0x8     0       1      0x55cc50aa67ba         0       279       289    16572         1  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0            0.00%   13.32%   14.72%    0.00%                 0xc     0       1      0x55cc50aa67ba         0       548       406    11974         2  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0            0.00%   10.36%   12.93%    0.00%                0x10     0       1      0x55cc50aa67ba         0       423       299    10135         1  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0            0.00%    9.82%   12.42%    0.00%                0x14     0       1      0x55cc50aa67ba         0       407       288     9638         1  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0            0.00%   12.72%   14.89%    0.00%                0x18     0       1      0x55cc50aa67ba         0       546       410    11897         1  [.] std::__atomic_base<int>::o  ceph_test_c2c  atomic_base.h:548   0 On 18/03/2021 18:25, Loïc Dachary wrote: -- Loïc Dachary, Artisan Logiciel Libre

Hi Joe, /proc/cpuinfo reads: model name      : Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz running: Linux enough 5.10.0-5-amd64 #1 SMP Debian 5.10.24-1 (2021-03-19) x86_64 GNU/Linux I'm not sure why the tests confused it with an AMD though. Do you ? It is *not* a server though: it is a laptop that I wiped out and reinstalled from scratch to use it for the duration of the test with no interference. Are laptops somehow immune to the problem? Cheers On 04/04/2021 17:12, jmario(a)redhat.com wrote: -- Loïc Dachary, Artisan Logiciel Libre

Hi Loic: Laptops are not immune to cacheline contention, but in my experience larger systems, with more cpus creates a better environment to see the contention. With more threads of execution (OSDs?) running on more cpus, the cacheline contention and tugging shows up much better. I was under the impression that those larger environments are similar to what Ceph is running in today where you're trying to minimize that contention. We can try it first on your laptop if that's all you have, and we'll see what it shows us. (But do try CentOS-8 instead of Debian). Do you want to go that route first? Jore

I uploaded the /boot/config-5.10.0-5-amd64 file to dropbox.redhat.com : it looks like perf is compiled in: # # Kernel Performance Events And Counters # CONFIG_PERF_EVENTS=y and all the options below are =y (no =n). Maybe a module should be loaded ? On 04/04/2021 17:54, Loïc Dachary wrote: -- Loïc Dachary, Artisan Logiciel Libre

On Sun, Apr 4, 2021 at 2:24 PM Joe Mario &lt;jmario(a)redhat.com&gt; wrote: Just to clarify, I was referring to my assumption about your coding differences between the "with" and "without" sharding versions of Ceph. Joe

Hi Loïc: On Sun, Apr 4, 2021 at 4:14 PM Loïc Dachary &lt;loic(a)dachary.org&gt; wrote: Here are two ways you can get the cacheline size. One is by reading /sys/devices/system/cpu/cpu0/cache/index0/coherency_line_size Another is with: gcc -DLEVEL1_DCACHE_LINESIZE=`getconf LEVEL1_DCACHE_LINESIZE` ... Another is with: grep -m1 cache_alignment /proc/cpuinfo Most often it's 64 bytes. I believe the power cpus are 128 bytes. Itanium was 128 bytes. However, even on the X86 platforms where the cacheline size is 64 bytes, it's very often a good idea to pad your hot locks or hot data items out to 128 bytes (e.g. 2 cachelines instead of 1). The reason is this: By default when Intel processors fetch a cacheline of data, the cpu will gratuitously fetch the next cacheline, just in case you need it. However if that next cacheline is a different hot cacheline, the last thing you need is invalidate it with gratuitous writes. We have seen performance problems due to this, and the resolution was to pad the hot locks and variables out to 128 bytes. Some of the big database vendors pad out to 128 bytes because of this as well. I looked at the 2nd tar.gz file that you uploaded (ceph-c2c-jmario-2021-04-04-22-13.tar.gz ). As expected, the "without-sharding" case looked like it did earlier. However, in the "with-sharding" case, it didn't even look like your ceph_test_c2c program was even running. I even dumped the raw samples from the perf.data file and didn't see any loads or stores from the program. Can you double check that it ran correctly? Joe

The version with sharding to avoid cacheline contention is indeed faster (about 5 times faster). I modified the test program to verify that it is consistently at least 2x faster. This is deliberately conservative: the goal is to guard against  a regression that would break the optimization entirely rather than trying to fine tune the optimization. There was such a regression in Ceph for a long time (fixed earlier this year) and it would be good if it does not happen again. In addition the test should also verify that the optimization actually relates to cacheline contention. If I understand correctly, the latest output I sent you shows that the non optimized version uses only one variable and there is a cacheline contention reported by perf. c2c The optimized version however has no cacheline contention at all, which is the intended effect of the optimization. Is my reasoning correct so far? On 05/04/2021 09:27, Loïc Dachary wrote: from the perf.data file and didn't see any loads or stores from the program.  Can you double check that it ran correctly? -- Loïc Dachary, Artisan Logiciel Libre

On Mon, Apr 5, 2021 at 3:27 AM Loïc Dachary &lt;loic(a)dachary.org&gt; wrote: Hi Loïc: Your new sharding version looks much better. I do not see any cacheline contention at all. Here's some insight into the difference. The atomic update you're doing to the lock-variable has to both read and write the lock-variable and does it while it has the cacheline for the lock-variable locked. The cpu doing that atomic instruction needs to first get ownership of the cacheline. If no other threads of execution are also trying to get ownership of that cacheline, then ownership is granted rather quickly. If, however, there are many other threads of execution trying to get ownership of that cacheline, then all those cpus must "get in line" and wait their turn. In your "without-sharding" case, the average number of machine cycles needed for the atomic instruction to gain ownership of the cacheline was 751 machine cycles. In the "with-sharding" case, it dropped to 84 machine cycles. Understand, however, the above numbers are not perfectly accurate. That's because perf was instructed to ignore any load instructions that took faster than 70 machine cycles to complete. The reasoning is at those low levels of machine cycles, there is no contention, so why burden the perf tool execution and data collection with the extra processing of fast loads that aren't relevant to finding cacheline contention. I mention this for completeness. The drop from 751 to 84 machine cycles is significant. Do you have something in your code to guarantee that nothing else resides in the same aligned "128 byte 2-cacheline block" as your locks? Joe

On Mon, Apr 5, 2021 at 10:38 AM Loïc Dachary &lt;loic(a)dachary.org&gt; wrote: [snip] In your test case, you avoided all cacheline contention by putting every lock into its own cacheline. In practice, however, multiple threads need to be contending for the same locks to modify shared data. The goal is to look at your application's hottest contended cachelines to see if that contention can be minimized. Some of the ways to do that include: 1) Seeing if the number of accesses to that line can be minimized, especially the writers. 2) Making sure multiple hot data variables don't share the same cacheline. 3) Looking to see if the accesses to the hot cachelines are coming from the same numa node as where the hot data lives. This isn't always possible, but it's good to examine it. Doing it for this test case was somewhat trivial, albeit fragile. That's because the "without-sharding" version only had one contended cacheline, and the "with-sharding" version had no contended cacheline. Because of that, I was able to dump the raw data and add up the load latencies for the test program's load instructions. The steps I used were: # perf script -i perf_c2c_a_all_user.data > t.script -f # grep ceph_test_c2c t.script |grep ldlat |sed -e 's/^.*LCK//' |awk '{s+=$2;c+=1}END {print s " " c " " s/c}' However the above simple script won't work when there's more than one hot cacheline involved. You can still find the data you want, it just gets more complicated. Plus, the real value in perf c2c is not just seeing the load latencies, but rather to learn everything about the contention which will then help guide how to minimize it. It provides a lot of insight into what's happening. How does this approach sound? 1) You set up Ceph to run on a bigger multi-node server with fast storage. 2) Run the attached script, which is just an updated version of the script you've been running. 3) Then we can set up a shared video call where I can walk you through the perf c2c output pointing out all the key pieces of information. 4) With the insight from "3" above, you can then decide what you might want to automate and how it might be done. Does that sound reasonable? See the attached "run_c2c_ceph2.sh" script. Joe

Hi Loïc On Sat, Apr 24, 2021 at 2:22 PM Loïc Dachary &lt;loic(a)dachary.org&gt; wrote: It is very easy to get lost. When I shared that "perf script" output with you, I never expected that you'd be using it. The output of "perf script" is pretty cryptic and not well documented. When I used the "perf script" command, it only worked because of your very unique test case, which had only one cacheline being contended. I have never used it before because no other program I ever looked at only had one cacheline with contention. Can that "perf script" technique be used in a larger program with more contended cachelines? Perhaps, but it wouldn't be my preferred approach. I did not followup on your suggestion to run perf on a running Ceph cluster But when I look at your data, I still only see the small test case. I assumed you would be running the c2c script on a large fast server where Ceph was under a heavy real workload. Is that something you can do? The risk of the small test case is that it may not represent the contention situations in the larger Ceph codebase. How about if we try this approach: 1) You run my c2c script on a large fast server running Ceph in a heavy workoad. I expect to see many cachelines being contended. 2) I analyze the results, and send back my findings. 3) We then have an interactive meeting where I can walk you through what I found and how I found it. 4) Then from step 4, you'll be able to better understand how you can create a test or tests that you can integrate into the Ceph test suite. Does the above make sense? Joe