Den tors 4 juni 2020 kl 16:29 skrev David Orman <ormandj@corenode.com>:

* bluestore: common/options.cc: disable bluefs_preextend_wal_files <-- from 15.2.3 changelogs. There was a bug which lead to issues on OSD

Given that preextended WAL files was mentioned as a speed increasing feature in nautilus 14.2.3 release notes, are nautilus clusters in danger or just Octopus?

restart, and I believe this was the attempt at mitigation until a proper bugfix could be put into place. I suspect this might be the cause of the symptoms you're seeing.

https://tracker.ceph.com/issues/45613 https://github.com/ceph/ceph/pull/35293

-- May the most significant bit of your life be positive.

Hi Stephan, We recently ran a set of 3-sample tests looking at 2OSD/NVMe vs 1 OSD/NVMe RBD performance on Nautilus, Octopus, and Master on some of our newer performance nodes with Intel P4510 NVMe drives. Those tests use the librbd fio backend.  We also saw similar randread and seq write performance increases but did not see a performance regression with 4KB random writes like you did.  In fact Octopus was significantly faster than Nautilus (but master regressed a little vs octopus).  We expect it to be significantly faster too as we improved the way the bluestore caches work and it's consistently shown gains for us.  Here are the most recent test results: https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4Ror... Having said that, this is the second report I've gotten regarding performance regression in Octopus so there could be something going on that we are missing.  If possible, could you run gdbpmp against one of your OSDs during the test?  That might help us figure out why it's slow.  Otherwise some other things to look at: 1) If this is a large dataset, see if increasing the osd_memory_target helps.  onode cache misses really hurt us and can increase latency and hurt IOPS.  Now that Adam's column family sharding PR has merged in master we have two complimentary PRs the both help reduce OSD memory consumption for caching onodes.  For now you might see higher performance if you can afford to give the OSDs more memory. 2) Check to see if the CPUs are being kept in a high power state.  The transition can cause higher latency and perversely the less CPU you use the more likely the CPU is to drop into a low power state resulting in higher latency and worse performance, especially if it ends up thrashing between power states. 3) Lately I haven't seen the kv sync thread acting as a hard bottleneck during 4KB random writes, but it still could be if you have a low clocked processor (especially in a power saving state).  This is still an area to look carefully at if performance is low. 4) the bluefs_buffered_io change was the other thing I suspected but it sounds like you've already tested that.  never-the-less it would be good to see if IOs are backing up.  If you can get a wall clock profile with gdbpmp you might be able to tell if io_submit is blocking.  iostat or collectl can also probably tell you if the device queue is backing up. Hope this gives some ideas to start out! Thanks, Mark On 6/4/20 10:07 AM, Stephan wrote:

Thanks for your fast reply! We just tried all four possible combinations of bluefs_preextend_wal_files and bluefs_buffered_io, but the write-iops in test "usecase1" remain the same. By the way bluefs_preextend_wal_files has been false in 14.2.9 (as in 15.2.3). Any other ideas?

David Orman wrote:

...

* bluestore: common/options.cc: disable bluefs_preextend_wal_files <-- from 15.2.3 changelogs. There was a bug which lead to issues on OSD restart, and I believe this was the attempt at mitigation until a proper bugfix could be put into place. I suspect this might be the cause of the symptoms you're seeing.

https://tracker.ceph.com/issues/45613 https://github.com/ceph/ceph/pull/35293

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Mark, thanks for your comprehensive response! Our tests are basically matching the linked results (we are testing with 2 OSDs/NVMe and fio/librbd too, but having a much smaller setup). Sometimes we see smaller or higher improvements from Nautilus to Octupus but it is similar. Only the random write iops are the other way round, namely a lot slower in our setup … Meanwhile we have gone through some more testing: @1) Increasing osd_memory_target from the default (which ist 4GB as far as we know) to 16GB doesn't change the results. @2/3) The CPUs are configured for high performance in the BIOS and we also ensured that it is set in the kernel as well (governor performance). Each node in our test-setup has one Intel E2690-v3 with 12/24 cores/threads running constantly at 3,1GHz. @4) Yes, we have tested bluefs_buffered_io without success. We did some profiling using gdbpmp, collecting 100 samples shows that 0.5%-1% of the time is spent in io_submit. There is an extrem performance impact when profiling (reducing iops to several hundreds operations/second), therefore we are uncertain if this is a relevant information. Can we improve the profiling (we used gdbpmp.py -p … -n 100 -m bstore_kv_sync,bstore_kv_final -o … like in the example on github)? We gladly provide the sample data collected if this could be helpful. Furthermore we checked iostats, which seems to be okay (w_await most times below 1). @5) We have set noscrub and norebalance as well as disabled the automated scaling of the pg count during all our tests. As the results are reproducible when switching between Nautilus and Octopus, there must clearly be something going on in Octopus. Maybe this only affects very small setups like ours? As far as we see you have been testing with 8 nodes/64 NVMe total, where our setup only consists of 3 nodes with one NVMe each. Kind regards Stephan

Hi Mark and others, last week we have finally been able to solve the problem. We are using Gentoo on our test cluster and as it turned out the official Ebuilds are not setting CMAKE_BUILD_TYPE=RelWithDebInfo, which alone caused the performance degradation we have been seeing after upgrading to Octopus. After patching the Ebuild we are now getting the same results as under Centos 8 using the official RPMs for 15.2.3, which we set up temporarily on the same hardware in order to narrow down the cause of the problem. As far as we understand, the Ebuilds for Nautilus are not setting CMAKE_BUILD_TYPE=RelWithDebInfo too. So we retested Nautilus built with CMAKE_BUILD_TYPE=RelWithDebInfo and compared the results to a build using the unpatched ebuild. But interestingly here we have been seeing hardly any difference, it seems to affect the performance of Octopus in particular at least in our small setup. We reported the issue in the Gentoo Bugtracker: https://bugs.gentoo.org/733316 The results we are now seeing with Octopus are in general as good or better compared to Nautilus. You can find them here: https://docs.google.com/spreadsheets/d/13XH3Uuvcq16rrEMp88_Lb-vJfkwjJCDYlBDR... We are seeing performance improvements by upgrading to Octopus, which are comparable to the results linked below, where you have been testing Nautilus vs. Octopus vs. Master with 8 Nodes and 64 NVMe, but to our surprise only when testing with a single client. When testing with nine clients, we are seeing a massive performance boost with sequential 4k writes, all other results are fairly the same as with Nautilus. Not sure why, but maybe the improvements only affect larger deployments like yours. On the other hand when comparing with your results, we think that our small cluster was performing quite well with Nautilus already. The only thing which is strange: We are losing about 60% performance with a single client and sequential 4k reads (and about 20% with nine clients) by upgrading to Octopus, which has not been the case in your tests, as far as we see. We have tested this several times, it's reproducible and consistent and it does not seem to be related to the issue with not setting CMAKE_BUILD_TYPE. Just wanted to share these results, as it might be interesting for you. Thank you very much for your help.

Based on v15.2.2, 5 storage node(nvme:OSD=1:2, optane as rocksdb backend) 5client, test case: fio, 20image, 4K Randread/randwrite 4KRR 4KRW default 760700 262500 PR34363 1185500 254300 disable-osdnuma 772600 340200 PR34363+disable_osdnuma 1364200 384700 Note: PR34363: https://github.com/ceph/ceph/pull/34363 dislabe_osdnumae: set osd_numa_auto_affinity && osd_numa_prefer_iface to false.