We have deployed a small test cluster consisting of three nodes. Each node is running a mon/mgr and two osds (Samsung PM983 3,84TB NVMe split into two partitions), so six osds in total. We started with Ceph 14.2.7 some weeks ago (upgraded to 14.2.9 later) and ran different tests using fio against some rbd volumes in order to get an overview what performance we could expect. The configuration is unchanged compared to the defaults, we only set several debugging options to 0/0. Yesterday we upgraded the whole cluster following the upgrade guidelines to Ceph 15.2.3, which worked without any problems so far. Nevertheless when running the same tests as before with Ceph 14.2.9, we are seeing some clear degradations in write-performance (beside some performance improvements, which shall also be mentioned). Here the results of concern (each with the relevant fio settings used): Test "read-latency-max" (rw=randread, iodepth=64, bs=4k) read_iops: 32500 -> 87000 Test "write-latency-max" (rw=randwrite, iodepth=64, bs=4k) write_iops: 22500 -> 11500 Test "write-throughput-iops-max" (rw=write, iodepth=64, bs=4k) write_iops: 7000 -> 14000 Test "usecase1" (rw=randrw, bssplit=4k/40:8k/5:16k/20:32k/5:64k/10:128k/10:256k/,4k/50:8k/20:16k/20:32k/5:64k/2:128k/:256k/, rwmixread=1, rate_process=poisson, iodepth=64) write_iops: 21000 -> 8500 Test "usecase1-readonly" (rw=randread, bssplit=4k/40:8k/5:16k/20:32k/5:64k/10:128k/10:256k/, rate_process=poisson, iodepth=64) read_iops: 28000 -> 58000 The last two tests represent a typical use case on our systems. Therefore we are especially concerned by the drop in performance from 21000 w/ops to 8500 w/ops (about 60%) after upgrading to Ceph 15.2.3. We ran all tests several times, the values are averaged over all iterations and fairly consistent and reproducible. We even tried wiping the whole cluster, downgrading to Ceph 14.2.9 again, setting up a new cluster/pool, running the tests and upgrading to Ceph 15.2.3 again. The tests have been performed on one of the three cluster nodes using a 50G rbd volume, which had been prefilled with random data before each test-run. Have any changes been introduced with Octopus that could explain the observed changes in performance? What we already tried: - Disabling rbd cache - Reverting rbc cache policy to writeback (default in 14.2) - Setting rbd io scheduler to none - Deploying a fresh cluster starting with Ceph 15.2.3 Kernel is 5.4.38 … I don't know if some other system specs would be helpful besides the already mentioned (since we are talking about a relative change in performance after upgrading Ceph without any further changes) - if so, please let us know. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Hi Stephan, We recently ran a set of 3-sample tests looking at 2OSD/NVMe vs 1 OSD/NVMe RBD performance on Nautilus, Octopus, and Master on some of our newer performance nodes with Intel P4510 NVMe drives. Those tests use the librbd fio backend.  We also saw similar randread and seq write performance increases but did not see a performance regression with 4KB random writes like you did.  In fact Octopus was significantly faster than Nautilus (but master regressed a little vs octopus).  We expect it to be significantly faster too as we improved the way the bluestore caches work and it's consistently shown gains for us.  Here are the most recent test results: https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4Ro… Having said that, this is the second report I've gotten regarding performance regression in Octopus so there could be something going on that we are missing.  If possible, could you run gdbpmp against one of your OSDs during the test?  That might help us figure out why it's slow.  Otherwise some other things to look at: 1) If this is a large dataset, see if increasing the osd_memory_target helps.  onode cache misses really hurt us and can increase latency and hurt IOPS.  Now that Adam's column family sharding PR has merged in master we have two complimentary PRs the both help reduce OSD memory consumption for caching onodes.  For now you might see higher performance if you can afford to give the OSDs more memory. 2) Check to see if the CPUs are being kept in a high power state. The transition can cause higher latency and perversely the less CPU you use the more likely the CPU is to drop into a low power state resulting in higher latency and worse performance, especially if it ends up thrashing between power states. 3) Lately I haven't seen the kv sync thread acting as a hard bottleneck during 4KB random writes, but it still could be if you have a low clocked processor (especially in a power saving state).  This is still an area to look carefully at if performance is low. 4) the bluefs_buffered_io change was the other thing I suspected but it sounds like you've already tested that.  never-the-less it would be good to see if IOs are backing up.  If you can get a wall clock profile with gdbpmp you might be able to tell if io_submit is blocking.  iostat or collectl can also probably tell you if the device queue is backing up. Hope this gives some ideas to start out! Thanks, Mark On 6/4/20 10:07 AM, Stephan wrote: > Thanks for your fast reply! We just tried all four possible > combinations of bluefs_preextend_wal_files and bluefs_buffered_io, > but the write-iops in test "usecase1" remain the same. By the way  > bluefs_preextend_wal_files has been false in 14.2.9 (as in 15.2.3). > Any other ideas? > > David Orman wrote: >> * bluestore: common/options.cc: disable bluefs_preextend_wal_files  <-- >> from 15.2.3 changelogs. There was a bug which lead to issues on OSD >> restart, and I believe this was the attempt at mitigation until a >> proper >> bugfix could be put into place. I suspect this might be the cause of >> the >> symptoms you're seeing. >> >> https://tracker.ceph.com/issues/45613 >> https://github.com/ceph/ceph/pull/35293 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

Hi Mark, thanks for your comprehensive response! Our tests are basically matching the linked results (we are testing with 2 OSDs/NVMe and fio/librbd too, but having a much smaller setup). Sometimes we see smaller or higher improvements from Nautilus to Octupus but it is similar. Only the random write iops are the other way round, namely a lot slower in our setup … Meanwhile we have gone through some more testing: @1) Increasing osd_memory_target from the default (which ist 4GB as far as we know) to 16GB doesn't change the results.

@2/3) The CPUs are configured for high performance in the BIOS and we also ensured that it is set in the kernel as well (governor performance). Each node in our test-setup has one Intel E2690-v3 with 12/24 cores/threads running constantly at 3,1GHz.

@4) Yes, we have tested bluefs_buffered_io without success. We did some profiling using gdbpmp, collecting 100 samples shows that 0.5%-1% of the time is spent in io_submit. There is an extrem performance impact when profiling (reducing iops to several hundreds operations/second), therefore we are uncertain if this is a relevant information. Can we improve the profiling (we used gdbpmp.py -p … -n 100 -m bstore_kv_sync,bstore_kv_final -o … like in the example on github)? We gladly provide the sample data collected if this could be helpful. Furthermore we checked iostats, which seems to be okay (w_await most times below 1).

@5) We have set noscrub and norebalance as well as disabled the automated scaling of the pg count during all our tests. As the results are reproducible when switching between Nautilus and Octopus, there must clearly be something going on in Octopus. Maybe this only affects very small setups like ours? As far as we see you have been testing with 8 nodes/64 NVMe total, where our setup only consists of 3 nodes with one NVMe each.