And to add some references, there is a PR on hold here: https://github.com/ceph/ceph/pull/38044 which links some relevant trackers entries. Outside of large block.db removals (e.g. from backfilling or snap trimming) we didn't notice a huge difference -- though that is not conclusive. There are several PG removal optimizations in the pipeline which hopefully fix the issues in a different way, rather than needing buffered io. -- dan On Mon, Jan 11, 2021 at 5:20 PM Mark Nelson <mnelson@redhat.com> wrote:

Hi Robert,

We are definitely aware of this issue. It appears to often be related to snap trimming and we believe possibly related to excessive thrashing of the rocksdb block cache. I suspect that when bluefs_buffered_io is enabled it hides the issue and people don't notice the problem, but that might be related to why we see the other issue with the kernel with rgw workloads. I would recommend that if you didn't see issues with bluefs_buffered_io enabled, you can re-enable it and periodically check to make sure you aren't hitting issues with kernel swap. Unfortunately we are sort of between a rock and a hard place on this one until we solve the root cause.

Right now we're looking at trying to reduce thrashing in the rocksdb block cache(s) by splitting up onode and omap (and potentially pglog and allocator) block cache into their own distinct entities. My hope is that we can finesse the situation so that the overall system page cache is no longer required to avoid execessive reads assuming enough memory has been assigned to the osd_memory_target.

Mark

On 1/11/21 9:47 AM, Robert Sander wrote:

...

Hi,

bluefs_buffered_io was disabled in Ceph version 14.2.11.

The cluster started last year with 14.2.5 and got upgraded over the year now running 14.2.16.

The performance was OK first but got abysmal bad at the end of 2020.

We checked the components and HDDs and SSDs seem to be fine. Single disk benchmarks showed performance according the specs.

Today we (re-)enabled bluefs_buffered_io and restarted all OSD processes on 248 HDDs distributed over 12 nodes.

Now the benchmarks are fine again: 434MB/s write instead of 60MB/s, 960MB/s read instead of 123MB/s.

This setting was disabled in 14.2.11 because "in some test cases it appears to cause excessive swap utilization by the linux kernel and a large negative performance impact after several hours of run time." We have to monitor if this will happen in our cluster. Is there any other negative side effect currently known?

Here are the rados bench values, first with bluefs_buffered_io=false, then with bluefs_buffered_io=true:

Bench Total Total Write Object Band Stddev Max Min Average Stddev Max Min Average Stddev Max Min time writes Read size width Bandwidth IOPS Latency (s) run reads size (MB/sec) made false write 33,081 490 4194304 4194304 59,2485 71,3829 264 0 14 17,8702 66 0 1,07362 2,83017 20,71 0,0741089 false seq 15,8226 490 4194304 4194304 123,874 30 46,8659 174 0 0,51453 9,53873 0,00343417 false rand 38,2615 2131 4194304 4194304 222,782 55 109,374 415 0 0,28191 12,1039 0,00327948 true write 30,4612 3308 4194304 4194304 434,389 26,0323 480 376 108 6,50809 120 94 0,14683 0,07368 0,99791 0,0751249 true seq 13,7628 3308 4194304 4194304 961,429 240 22,544 280 184 0,06528 0,88676 0,00338191 true rand 30,1007 8247 4194304 4194304 1095,92 273 25,5066 313 213 0,05719 0,99140 0,00325295

Regards

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io