On 10/01/2020 21.54, Radoslaw Zarzynski wrote:
Hi Roman,
First thing that catches my eye is that for small blocks there is no big difference at all, but as the block increases, crimsons iops starts to decline. Can it be the transport issue? Can be tested as well. This is a known issue with the Seastar's POSIX-based network stack. As Kefu pointed out, even large payloads are retrieved from kernel with multiple small, fixed-size chunks. That's a matter of how
On Thu, Jan 9, 2020 at 2:51 PM Roman Penyaev <rpenyaev@suse.de> wrote: the internal interfaces were shaped. My personal impression is their design favors the native stack / DPDK while avoiding differentiated behavior among the stacks (likely to not surprise developers).
The goal is not to require different optimization techniques or code paths for the two stacks. And it is a challenge to do this well. I guess we need a way to provide the state machine to seastar::input_stream so that it can provide the expected data layout to data_source. Maybe we need to bypass input_stream completely - it tries to buffer data by itself but here the application knows better. So perhaps we need to interact with data_source instead. The application would provide the buffers for the data_source, and seastar would take care of the copying if the data_source cannot use the buffer directly (the native stack case).
Moreover, crimson-osd imposes on Seastar additional memcpy to reconcile those tiny chunks into a flat buffer.
Here is a more detailed gist: https://gist.github.com/rzarzynski/a1d67dc39b0ef4d49cb522179b1f3c89.
There are branches (for both Seastar and crimson) with PoC for the "input buffer factory" that targets those issues. Performance comparison is here: https://gist.github.com/rzarzynski/ad0aaa80b26603bc1a803ce0d209ac87.
Also, when narrowing the comparison to async-msgr vs crimson-msgr (with ibf) I wouldn't expect too much of a difference. In read tests we're observing pretty similar IPC for both crimson-osd and msgr-worker-n (single thread profiling). The thing that might change a lot is the native stack. In Intel's testing it significantly (up to 30-40% IIRC) improved IPC of crimson-msgr. Glued with good SPDK support in Seastore it might draw the POSIX stack (and thus the need for ibf) a bit obsolete.
Quick note on the saturation: please be careful when judging it with top, pidstat or even perf stat. In contrast to the legacy OSD, Seastar does busy-wait for awhile. This greatly exaggerates the CPU utilisation for modest workloads. And yes, **crimson is all about the computational efficiency**. We're much more interested in cycles/op than in raw IOPS, to be honest.
Regards, Radek _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io