Hi Stephan, We recently ran a set of 3-sample tests looking at 2OSD/NVMe vs 1 OSD/NVMe RBD performance on Nautilus, Octopus, and Master on some of our newer performance nodes with Intel P4510 NVMe drives. Those tests use the librbd fio backend.  We also saw similar randread and seq write performance increases but did not see a performance regression with 4KB random writes like you did.  In fact Octopus was significantly faster than Nautilus (but master regressed a little vs octopus).  We expect it to be significantly faster too as we improved the way the bluestore caches work and it's consistently shown gains for us.  Here are the most recent test results: https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4Ro… Having said that, this is the second report I've gotten regarding performance regression in Octopus so there could be something going on that we are missing.  If possible, could you run gdbpmp against one of your OSDs during the test?  That might help us figure out why it's slow.  Otherwise some other things to look at: 1) If this is a large dataset, see if increasing the osd_memory_target helps.  onode cache misses really hurt us and can increase latency and hurt IOPS.  Now that Adam's column family sharding PR has merged in master we have two complimentary PRs the both help reduce OSD memory consumption for caching onodes.  For now you might see higher performance if you can afford to give the OSDs more memory. 2) Check to see if the CPUs are being kept in a high power state. The transition can cause higher latency and perversely the less CPU you use the more likely the CPU is to drop into a low power state resulting in higher latency and worse performance, especially if it ends up thrashing between power states. 3) Lately I haven't seen the kv sync thread acting as a hard bottleneck during 4KB random writes, but it still could be if you have a low clocked processor (especially in a power saving state).  This is still an area to look carefully at if performance is low. 4) the bluefs_buffered_io change was the other thing I suspected but it sounds like you've already tested that.  never-the-less it would be good to see if IOs are backing up.  If you can get a wall clock profile with gdbpmp you might be able to tell if io_submit is blocking.  iostat or collectl can also probably tell you if the device queue is backing up. Hope this gives some ideas to start out! Thanks, Mark On 6/4/20 10:07 AM, Stephan wrote: > Thanks for your fast reply! We just tried all four possible > combinations of bluefs_preextend_wal_files and bluefs_buffered_io, > but the write-iops in test "usecase1" remain the same. By the way  > bluefs_preextend_wal_files has been false in 14.2.9 (as in 15.2.3). > Any other ideas? > > David Orman wrote: >> * bluestore: common/options.cc: disable bluefs_preextend_wal_files  <-- >> from 15.2.3 changelogs. There was a bug which lead to issues on OSD >> restart, and I believe this was the attempt at mitigation until a >> proper >> bugfix could be put into place. I suspect this might be the cause of >> the >> symptoms you're seeing. >> >> https://tracker.ceph.com/issues/45613 >> https://github.com/ceph/ceph/pull/35293 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

Hi Mark, thanks for your comprehensive response! Our tests are basically matching the linked results (we are testing with 2 OSDs/NVMe and fio/librbd too, but having a much smaller setup). Sometimes we see smaller or higher improvements from Nautilus to Octupus but it is similar. Only the random write iops are the other way round, namely a lot slower in our setup … Meanwhile we have gone through some more testing: @1) Increasing osd_memory_target from the default (which ist 4GB as far as we know) to 16GB doesn't change the results.

@2/3) The CPUs are configured for high performance in the BIOS and we also ensured that it is set in the kernel as well (governor performance). Each node in our test-setup has one Intel E2690-v3 with 12/24 cores/threads running constantly at 3,1GHz.

@4) Yes, we have tested bluefs_buffered_io without success. We did some profiling using gdbpmp, collecting 100 samples shows that 0.5%-1% of the time is spent in io_submit. There is an extrem performance impact when profiling (reducing iops to several hundreds operations/second), therefore we are uncertain if this is a relevant information. Can we improve the profiling (we used gdbpmp.py -p … -n 100 -m bstore_kv_sync,bstore_kv_final -o … like in the example on github)? We gladly provide the sample data collected if this could be helpful. Furthermore we checked iostats, which seems to be okay (w_await most times below 1).

@5) We have set noscrub and norebalance as well as disabled the automated scaling of the pg count during all our tests. As the results are reproducible when switching between Nautilus and Octopus, there must clearly be something going on in Octopus. Maybe this only affects very small setups like ours? As far as we see you have been testing with 8 nodes/64 NVMe total, where our setup only consists of 3 nodes with one NVMe each.