9 Jan
2020
9 Jan
'20
8:25 p.m.
** resending to this list as well, since did not know that ceph-devel@
is for kernel patches only
Hi folks,
I was curios to read some early performance benchmarks which compare
crimson-osd vs legacy-osd, but could not find any. So eventually
decided to do my own micro benchmarks in order to test transport
together with PG layer, avoiding any storage costs completely
(no reason to test memcpy of memstore which is the only available
objectstore for crimson). At least recalling all these ad brochures
of seastar which should bring performance on another level by doing
preemption in userspace, the difference should be already there and
visible in numbers.
And yes I'm aware that crimson is in development, but if basic
functionality is already supported (like write path), then I can
squeeze some numbers.
For all testing loads I run original rbd.fio, taken from fio/examples/,
of course changing only block size. Since this is a micro benchmark I
run only 1 osd cluster.
[global]
ioengine=rbd
clientname=admin
pool=rbd
rbdname=fio_test
rw=randwrite
#bs=4k
[rbd_iodepth32]
iodepth=32
-- Part 1, turn MemStore and cyan_store into null block
Testing memcpy is not interesting so in order to run any memstore with
'memstore_debug_omit_block_device_write=true' option set and skip all
writes I have to do a small tweak in order to start osd, namely I still
need to pass small writes and omit big ones which are sent by the
client,
something as the following:
- if (len > 0 && !local_conf()->memstore_debug_omit_block_device_write)
{
+
+ if (len > 0 &&
+ (!local_conf()->memstore_debug_omit_block_device_write ||
+ // We still want cluster meta-data to be saved, so pass only
small
+ // writes, expecting user writes will be >= 4k.
+ len < 4096)) {
*** BTW at the bottom you can find the whole patch with all debug
modifications made to deliver these numbers.
# legacy-osd, MemStore
MON=1 MDS=0 OSD=1 MGR=1 ../src/vstart.sh --memstore -n \
-o 'memstore_debug_omit_block_device_write=true'
4k IOPS=42.7k, BW=167MiB/s, Lat=749.18usec
8k IOPS=40.2k, BW=314MiB/s, Lat=795.03usec
16k IOPS=37.6k, BW=588MiB/s, Lat=849.12usec
32k IOPS=32.0k, BW=1000MiB/s, Lat=998.56usec
64k IOPS=25.5k, BW=1594MiB/s, Lat=1253.99usec
128k IOPS=17.5k, BW=2188MiB/s, Lat=1826.54usec
256k IOPS=10.1k, BW=2531MiB/s, Lat=3157.33usec
512k IOPS=5252, BW=2626MiB/s, Lat=6071.37usec
1m IOPS=2656, BW=2656MiB/s, Lat=12029.65usec
# crimson-osd, cyan_store
MON=1 MDS=0 OSD=1 MGR=1 ../src/vstart.sh --crimson --memstore -n \
-o 'memstore_debug_omit_block_device_write=true'
4k IOPS=40.2k, BW=157MiB/s, Lat=796.07usec
8k IOPS=37.1k, BW=290MiB/s, Lat=861.51usec
16k IOPS=32.9k, BW=514MiB/s, Lat=970.99usec
32k IOPS=26.1k, BW=815MiB/s, Lat=1225.78usec
64k IOPS=21.3k, BW=1333MiB/s, Lat=1498.92usec
128k IOPS=14.4k, BW=1795MiB/s, Lat=2227.07usec
256k IOPS=6143, BW=1536MiB/s, Lat=5203.70usec
512k IOPS=3776, BW=1888MiB/s, Lat=8464.79usec
1m IOPS=1866, BW=1867MiB/s, Lat=17126.36usec
First thing that catches my eye is that for small blocks there is no big
difference at all, but as the block increases, crimsons iops starts to
decline. Can it be the transport issue? Can be tested as well.
-- Part 2, complete writes immediately, even not leaving the transport
Would be great to avoid PG logic costs, exactly like we did for
objectstore,
i.e. the following question can be asked "how fast we can handle writes
and
complete them immediately from the transport callback and measure socket
read/write costs?". I introduced new option 'osd_immediate_completions'
and handle it directly from 'OSD::ms_[fast_]dispatch' function replying
with
success just immediately (for details see patch at the bottom).
# legacy-osd
MON=1 MDS=0 OSD=1 MGR=1 ../src/vstart.sh --memstore -n \
-o 'osd_immediate_completions=true'
4k IOPS=59.2k, BW=231MiB/s, Lat=539.68usec
8k IOPS=55.1k, BW=430MiB/s, Lat=580.44usec
16k IOPS=50.5k, BW=789MiB/s, Lat=633.03usec
32k IOPS=44.6k, BW=1394MiB/s, Lat=716.74usec
64k IOPS=33.5k, BW=2093MiB/s, Lat=954.60usec
128k IOPS=20.8k, BW=2604MiB/s, Lat=1535.01usec
256k IOPS=10.6k, BW=2642MiB/s, Lat=3026.19usec
512k IOPS=5400, BW=2700MiB/s, Lat=5920.86usec
1m IOPS=2549, BW=2550MiB/s, Lat=12539.40usec
# crimson-osd
MON=1 MDS=0 OSD=1 MGR=1 ../src/vstart.sh --crimson --memstore -n \
-o 'osd_immediate_completions=true'
4k IOPS=60.2k, BW=235MiB/s, Lat=530.95usec
8k IOPS=52.0k, BW=407MiB/s, Lat=614.21usec
16k IOPS=47.1k, BW=736MiB/s, Lat=678.41usec
32k IOPS=37.8k, BW=1180MiB/s, Lat=846.75usec
64k IOPS=26.6k, BW=1660MiB/s, Lat=1203.51usec
128k IOPS=15.5k, BW=1936MiB/s, Lat=2064.12usec
256k IOPS=7506, BW=1877MiB/s, Lat=4259.19usec
512k IOPS=3941, BW=1971MiB/s, Lat=8112.67usec
1m IOPS=1785, BW=1786MiB/s, Lat=17896.44usec
As a summary I can say that for me is quite surprising not to notice any
iops improvements on crimson side (not to mention the problem with
reading
of big blocks). Since I run only 1 osd on one particular load I admit
the
artificial nature of such tests (thus called micro benchmark), but then
on what cluster scale and what benchmark can I run to see some
improvements
of a new architecture?
Roman
---
src/common/options.cc | 4 ++++
src/crimson/os/cyan_store.cc | 6 +++++-
src/crimson/osd/ops_executer.cc | 4 ++--
src/crimson/osd/osd.cc | 20 ++++++++++++++++++++
src/os/memstore/MemStore.cc | 6 +++++-
src/osd/OSD.cc | 24 ++++++++++++++++++++++++
6 files changed, 60 insertions(+), 4 deletions(-)
diff --git a/src/common/options.cc b/src/common/options.cc
index d91827c1a803..769666d2955c 100644
--- a/src/common/options.cc
+++ b/src/common/options.cc
@@ -4234,6 +4234,10 @@ std::vector<Option> get_global_options() {
.set_default(false)
.set_description(""),
+ Option("osd_immediate_completions", Option::TYPE_BOOL,
Option::LEVEL_ADVANCED)
+ .set_default(false)
+ .set_description(""),
+
// --------------------------
// bluestore
diff --git a/src/crimson/os/cyan_store.cc b/src/crimson/os/cyan_store.cc
index f0749cb921f9..c05e0e40b721 100644
--- a/src/crimson/os/cyan_store.cc
+++ b/src/crimson/os/cyan_store.cc
@@ -463,7 +463,11 @@ int CyanStore::_write(const coll_t& cid, const
ghobject_t& oid,
return -ENOENT;
ObjectRef o = c->get_or_create_object(oid);
- if (len > 0 && !local_conf()->memstore_debug_omit_block_device_write)
{
+ if (len > 0 &&
+ (!local_conf()->memstore_debug_omit_block_device_write ||
+ // We still want cluster meta-data to be saved, so pass only
small
+ // writes, expecting user writes will be >= 4k.
+ len < 4096)) {
const ssize_t old_size = o->get_size();
o->write(offset, bl);
used_bytes += (o->get_size() - old_size);
diff --git a/src/crimson/osd/ops_executer.cc
b/src/crimson/osd/ops_executer.cc
index 13f6f086c4ea..a76fc6e206d8 100644
--- a/src/crimson/osd/ops_executer.cc
+++ b/src/crimson/osd/ops_executer.cc
@@ -431,8 +431,8 @@ OpsExecuter::execute_osd_op(OSDOp& osd_op)
default:
logger().warn("unknown op {}", ceph_osd_op_name(op.op));
- throw std::runtime_error(
- fmt::format("op '{}' not supported", ceph_osd_op_name(op.op)));
+ // Without that `fio examples/rbd.fio` hangs on exit
+ throw ceph::osd::operation_not_supported{};
}
}
diff --git a/src/crimson/osd/osd.cc b/src/crimson/osd/osd.cc
index ddd8742d1a74..737cc266766e 100644
--- a/src/crimson/osd/osd.cc
+++ b/src/crimson/osd/osd.cc
@@ -17,6 +17,7 @@
#include "messages/MOSDOp.h"
#include "messages/MOSDPGLog.h"
#include "messages/MOSDRepOpReply.h"
+#include "messages/MOSDOpReply.h"
#include "messages/MPGStats.h"
#include "os/Transaction.h"
@@ -881,6 +882,25 @@ seastar::future<> OSD::committed_osd_maps(version_t
first,
seastar::future<> OSD::handle_osd_op(ceph::net::Connection* conn,
Ref<MOSDOp> m)
{
+
+ //
+ // Immediately complete requests even without leaving the transport
+ //
+ if (local_conf().get_val<bool>("osd_immediate_completions")) {
+ m->finish_decode();
+
+ for (auto op : m->ops) {
+ if (op.op.op == CEPH_OSD_OP_WRITE &&
+ // Complete big writes only
+ op.op.extent.length >= 4096) {
+
+ auto reply = make_message<MOSDOpReply>(m.get(), 0,
osdmap->get_epoch(),
+ CEPH_OSD_FLAG_ACK |
CEPH_OSD_FLAG_ONDISK, true);
+ return conn->send(reply);
+ }
+ }
+ }
+
shard_services.start_operation<ClientRequest>(
*this,
conn->get_shared(),
diff --git a/src/os/memstore/MemStore.cc b/src/os/memstore/MemStore.cc
index 05d16edb6cc0..265dc64c808d 100644
--- a/src/os/memstore/MemStore.cc
+++ b/src/os/memstore/MemStore.cc
@@ -1047,7 +1047,11 @@ int MemStore::_write(const coll_t& cid, const
ghobject_t& oid,
return -ENOENT;
ObjectRef o = c->get_or_create_object(oid);
- if (len > 0 && !cct->_conf->memstore_debug_omit_block_device_write) {
+ if (len > 0 &&
+ (!cct->_conf->memstore_debug_omit_block_device_write ||
+ // We still want cluster meta-data to be saved, so pass only
small
+ // writes, expecting user writes will be bigger than 4k.
+ len < 4096)) {
const ssize_t old_size = o->get_size();
o->write(offset, bl);
used_bytes += (o->get_size() - old_size);
diff --git a/src/osd/OSD.cc b/src/osd/OSD.cc
index 96aed0b706e3..796bf927126f 100644
--- a/src/osd/OSD.cc
+++ b/src/osd/OSD.cc
@@ -7223,6 +7223,30 @@ void OSD::ms_fast_dispatch(Message *m)
return;
}
+ //
+ // Immediately complete requests even without leaving the transport
+ //
+ if (g_conf().get_val<bool>("osd_immediate_completions") &&
+ m->get_type() == CEPH_MSG_OSD_OP) {
+ MOSDOp *osdop = static_cast<MOSDOp*>(m);
+
+ osdop->finish_decode();
+
+ for (auto op : osdop->ops) {
+ if (op.op.op == CEPH_OSD_OP_WRITE &&
+ // Complete big writes only
+ op.op.extent.length >= 4096) {
+ MOSDOpReply *reply;
+
+ reply = new MOSDOpReply(osdop, 0, osdmap->get_epoch(),
+ CEPH_OSD_FLAG_ACK |
CEPH_OSD_FLAG_ONDISK, true);
+ osdop->get_connection()->send_message(reply);
+ m->put();
+ return;
+ }
+ }
+ }
+
// peering event?
switch (m->get_type()) {
case CEPH_MSG_PING:
11 Jan
11 Jan
1:24 a.m.
Hi Roman,
On Thu, Jan 9, 2020 at 2:51 PM Roman Penyaev <rpenyaev(a)suse.de> wrote:
First thing that catches my eye is that for small
blocks there is no big
difference at all, but as the block increases, crimsons iops starts to
decline. Can it be the transport issue? Can be tested as well.
This is a known issue with the Seastar's POSIX-based network
stack. As Kefu pointed out, even large payloads are retrieved from
kernel with multiple small, fixed-size chunks. That's a matter of how
the internal interfaces were shaped. My personal impression is their
design favors the native stack / DPDK while avoiding differentiated
behavior among the stacks (likely to not surprise developers).
Moreover, crimson-osd imposes on Seastar additional memcpy to
reconcile those tiny chunks into a flat buffer.
Here is a more detailed gist:
https://gist.github.com/rzarzynski/a1d67dc39b0ef4d49cb522179b1f3c89.
There are branches (for both Seastar and crimson) with PoC for
the "input buffer factory" that targets those issues. Performance
comparison is here:
https://gist.github.com/rzarzynski/ad0aaa80b26603bc1a803ce0d209ac87.
Also, when narrowing the comparison to async-msgr vs crimson-msgr
(with ibf) I wouldn't expect too much of a difference. In read tests we're
observing pretty similar IPC for both crimson-osd and msgr-worker-n
(single thread profiling). The thing that might change a lot is the native
stack. In Intel's testing it significantly (up to 30-40% IIRC) improved
IPC of crimson-msgr. Glued with good SPDK support in Seastore
it might draw the POSIX stack (and thus the need for ibf) a bit obsolete.
Quick note on the saturation: please be careful when judging it with
top, pidstat or even perf stat. In contrast to the legacy OSD, Seastar
does busy-wait for awhile. This greatly exaggerates the CPU utilisation
for modest workloads. And yes, **crimson is all about the computational
efficiency**. We're much more interested in cycles/op than in raw IOPS,
to be honest.
Regards,
Radek
9:19 p.m.
Hi Radek,
Then taking into account everything you've said what load is a win
for crimson? What exactly should I run to notice that all shiny things
which seastar provides actually work and bring a value? (and I talk
about iops, because that's actually what matters for storage after all).
I just want to run a simple benchmark, which can show me a clear
difference in iops numbers between legacy and crimson osd.
What should I run?
--
Roman
On 2020-01-10 20:54, Radoslaw Zarzynski wrote:
Hi Roman,
On Thu, Jan 9, 2020 at 2:51 PM Roman Penyaev <rpenyaev(a)suse.de> wrote:
First thing that catches my eye is that for small
blocks there is no
big
difference at all, but as the block increases, crimsons iops starts to
decline. Can it be the transport issue? Can be tested as well.
This is a known issue with the Seastar's POSIX-based network
stack. As Kefu pointed out, even large payloads are retrieved from
kernel with multiple small, fixed-size chunks. That's a matter of how
the internal interfaces were shaped. My personal impression is their
design favors the native stack / DPDK while avoiding differentiated
behavior among the stacks (likely to not surprise developers).
Moreover, crimson-osd imposes on Seastar additional memcpy to
reconcile those tiny chunks into a flat buffer.
Here is a more detailed gist:
https://gist.github.com/rzarzynski/a1d67dc39b0ef4d49cb522179b1f3c89.
There are branches (for both Seastar and crimson) with PoC for
the "input buffer factory" that targets those issues. Performance
comparison is here:
https://gist.github.com/rzarzynski/ad0aaa80b26603bc1a803ce0d209ac87.
Also, when narrowing the comparison to async-msgr vs crimson-msgr
(with ibf) I wouldn't expect too much of a difference. In read tests
we're
observing pretty similar IPC for both crimson-osd and msgr-worker-n
(single thread profiling). The thing that might change a lot is the
native
stack. In Intel's testing it significantly (up to 30-40% IIRC) improved
IPC of crimson-msgr. Glued with good SPDK support in Seastore
it might draw the POSIX stack (and thus the need for ibf) a bit
obsolete.
Quick note on the saturation: please be careful when judging it with
top, pidstat or even perf stat. In contrast to the legacy OSD, Seastar
does busy-wait for awhile. This greatly exaggerates the CPU utilisation
for modest workloads. And yes, **crimson is all about the computational
efficiency**. We're much more interested in cycles/op than in raw IOPS,
to be honest.
Regards,
Radek
13 Jan
13 Jan
7:06 p.m.
Hi Roman,
On Sat, Jan 11, 2020 at 4:49 PM Roman Penyaev <rpenyaev(a)suse.de> wrote:
Then taking into account everything you've said
what load is a win
for crimson? What exactly should I run to notice that all shiny things
which seastar provides actually work and bring a value?
I usually go with 4 KB random reads using `rados bench`. For the main
path It should generate workload very similar to RBD while being much
more efficient generator than FIO. The former requires a fair amount of
CPUs which can be restricting when doing quick dev tests with multiple
instances (crimson requires at least two to saturate) on a HW with
limited CPU count.
(and I talk
about iops, because that's actually what matters for storage after all).
This requires clarification. IOPS from single OSD instance or from
a set of HW? I think that all finally matters is return-on-investment.
Number of processes inside a box is an implementation detail.
I just want to run a simple benchmark, which can show
me a clear
difference in iops numbers between legacy and crimson osd.
Sure. Currently we have the `perf check bot` doing `scripts/run-cbt.sh`
on `crimson-osd` to hunt for regressions. However, deploying
environment might be a bit time-consuming, so I quickly & roughly
compared both OSD implementations in the old school way:
https://gist.github.com/rzarzynski/9f92f951c929d4a4ed9a7a13ff156b71
All commands all there. I would be grateful for verifying the results.
Regards,
Radek
10:06 p.m.
On 2020-01-13 14:36, Radoslaw Zarzynski wrote:
[skip]
I do not understand. I talk about simple comparison metric for any
storage application - IOPS. Since both storage applications
(legacy-osd,
crimson-osd) share absolutely the same Ceph spec - that is a fair
choice.
For example we do not compare replication protocols (strong consistency
vs eventual or weak consistency), or client driven replication vs
primary
copy replication. These parts in our comparison are the same. But what
we compare is how fast message is received and reply is sent back.
Nothing
more. So what else if not IOPS?
If I were a game developer I would probably choose fps :)
Here is my:
https://gist.github.com/rouming/dcdca2d6e23fa7ae9041bbda28efd2a1
legacy-osd: ~130MB/s
crimson-osd: ~120MB/s
BTW prefill writes are also corresponding to reads on my machine:
legacy-osd: 122MB/s
crimson-osd: 94MB/s
What other loads I can try?
--
Roman
(and I talk
about iops, because that's actually what matters for storage after
all).
This requires clarification. IOPS from single OSD instance or from
a set of HW? I think that all finally matters is return-on-investment.
Number of processes inside a box is an implementation detail. I just want to
run a simple benchmark, which can show me a clear
difference in iops numbers between legacy and crimson osd.
Sure. Currently we have the `perf check bot` doing `scripts/run-cbt.sh`
on `crimson-osd` to hunt for regressions. However, deploying
environment might be a bit time-consuming, so I quickly & roughly
compared both OSD implementations in the old school way:
https://gist.github.com/rzarzynski/9f92f951c929d4a4ed9a7a13ff156b71
All commands all there. I would be grateful for verifying the results.
14 Jan
14 Jan
12:08 a.m.
Hi Roman,
On Mon, Jan 13, 2020 at 5:36 PM Roman Penyaev <rpenyaev(a)suse.de> wrote:
I do not understand. I talk about simple comparison
metric for any
storage application - IOPS. Since both storage applications
(legacy-osd,
crimson-osd) share absolutely the same Ceph spec - that is a fair
choice.
That way you're actually thinking about IOPS from an OSD instance
*disregarding how much HW resources it spends* to serve your
workload. This comparison ignores absolutely fundamental difference
in architecture:
* crimson-osd is single-threaded at the moment. It won't eat more
than 1 CPU core. That's by design.
* ceph-osd is multi-threaded. By default single instance has up to 16
`tp_osd_tp` and 3 `msgr-worker-n` threads. This translates into upper,
theoretical bound of 19 CPU cores. In practice it's of course much
lower but still far above than for crimson-osd.
Both implementations share the same restriction: amount invested on
hardware resources to run the cluster. How much IOPS you will get from
it is determined by the OSD's *computational efficiency*.
The goal is to maximize IOPS from fixed set of hardware OR, rephrased,
to minimize the hardware resources needed to provide a given amount
of IOPS.
The problem is awfully similar to the performance-per-watt metric and
CPU's power efficiency. Electrical / cooling power is scarce resource
just like number of CPU cores is in a Ceph cluster.
Regards,
Radek
12:25 a.m.
Put another way, consider the savings:
o If today you have to deploy dual-socket servers for enough cycles, especially with
dmcrypt and during recovery.
o Tomorrow you could deploy cost-effective servers with much less expensive single-socket
CPUs (and thus freedom from NUMA hassles)
alternately:
o Denser servers with more OSDs per node but the same number of sockets / cores
For larger clusters the CapEx efficiency will be substantial. Maybe a cluster that
economics previously limited to HDDs can now be all-SSD.
— aad
I do not understand. I talk about simple
comparison metric for any
storage application - IOPS. Since both storage applications
(legacy-osd,
crimson-osd) share absolutely the same Ceph spec - that is a fair
choice.
That way you're actually thinking about IOPS from an OSD instance
*disregarding how much HW resources it spends* to serve your
workload. This comparison ignores absolutely fundamental difference
in architecture:
* crimson-osd is single-threaded at the moment. It won't eat more
than 1 CPU core. That's by design.
* ceph-osd is multi-threaded. By default single instance has up to 16
`tp_osd_tp` and 3 `msgr-worker-n` threads. This translates into upper,
theoretical bound of 19 CPU cores. In practice it's of course much
lower but still far above than for crimson-osd.
Both implementations share the same restriction: amount invested on
hardware resources to run the cluster. How much IOPS you will get from
it is determined by the OSD's *computational efficiency*.
The goal is to maximize IOPS from fixed set of hardware OR, rephrased,
to minimize the hardware resources needed to provide a given amount
of IOPS.
The problem is awfully similar to the performance-per-watt metric and
CPU's power efficiency. Electrical / cooling power is scarce resource
just like number of CPU cores is in a Ceph cluster.
Regards,
Radek
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
1:18 a.m.
On 2020-01-13 19:38, Radoslaw Zarzynski wrote:
Hi Roman,
On Mon, Jan 13, 2020 at 5:36 PM Roman Penyaev <rpenyaev(a)suse.de> wrote:
Then probably it makes more sense to execute the same load but with
iodepth=1?
Otherwise it is not quite fair: legacy-osd is able to execute requests
in
parallel, but crimson-osd is not.
$ bin/rados bench -p test-pool 10 rand -t 1
legacy-osd: 34MB/s
crimson-osd: 53MB/s
At least this is fair and there is a noticeable difference.
Also I'm still curios how fast can be immediate completion of requests,
without leaving the transport layer and avoiding PG logic completely.
With 'osd_immediate_completions=true' option set (there is a patch in
the first email of this thread) the bandwidth is the same for both:
$ bin/rados bench -p test-pool 10 write -b 4096 --no-cleanup -t 1
legacy-osd: 63MB/s
crimson-osd: 63MB/s
Which, I would say, not quite impressive.
--
Roman
I do not understand. I talk about simple
comparison metric for any
storage application - IOPS. Since both storage applications
(legacy-osd,
crimson-osd) share absolutely the same Ceph spec - that is a fair
choice.
That way you're actually thinking about IOPS from an OSD instance
*disregarding how much HW resources it spends* to serve your
workload. This comparison ignores absolutely fundamental difference
in architecture:
* crimson-osd is single-threaded at the moment. It won't eat more
than 1 CPU core. That's by design.
* ceph-osd is multi-threaded. By default single instance has up to 16
`tp_osd_tp` and 3 `msgr-worker-n` threads. This translates into
upper,
theoretical bound of 19 CPU cores. In practice it's of course much
lower but still far above than for crimson-osd.
2:41 a.m.
On Mon, Jan 13, 2020 at 7:38 PM Radoslaw Zarzynski <rzarzyns(a)redhat.com> wrote:
* crimson-osd is single-threaded at the moment. It
won't eat more
than 1 CPU core. That's by design.
Quick supplement on that: actually there is a way to let single
crimson-osd instance to span multiple CPU cores while preserving
the shared-nothing design. Definitely the doors for it should be
kept open but – as it requires an extension to the RADOS
protocol – it would be preferred to not hurry up. IMHO we should
consider it after the seastore (BTW: persistent object store is
worth ~200 kcycles / op accordingly to Mark's testing).
Regards,
Radek
4:42 p.m.
On 2020-01-13 22:11, Radoslaw Zarzynski wrote:
On Mon, Jan 13, 2020 at 7:38 PM Radoslaw Zarzynski
<rzarzyns(a)redhat.com> wrote:
Am I right that you are talking about several connections between
a primary osd and a single client instance? At least I'm picturing
that each connection represents a software cpu (or how this thread,
which does scheduling, is called?) on osd side. Then I can imagine
that a request to a PG goes to one of the connections by simple
modulo operation (something like PG_id % Number_of_conns). So all
requests to a PG from all clients will be eventually handled by one
of the cpu threads. Something like that?
* crimson-osd is single-threaded at the moment.
It won't eat more
than 1 CPU core. That's by design.
Quick supplement on that: actually there is a way to let single
crimson-osd instance to span multiple CPU cores while preserving
the shared-nothing design. Definitely the doors for it should be
kept open but – as it requires an extension to the RADOS
protocol – it would be preferred to not hurry up. IMHO we should
consider it after the seastore (BTW: persistent object store is
worth ~200 kcycles / op accordingly to Mark's testing).
May I take a look on the link with numbers and what exactly persistent
object store you've mentioning?
--
Roman
15 Jan
15 Jan
1:35 a.m.
On Tue, Jan 14, 2020 at 12:13 PM Roman Penyaev <rpenyaev(a)suse.de> wrote:
Am I right that you are talking about several
connections between
a primary osd and a single client instance? At least I'm picturing
that each connection represents a software cpu (or how this thread,
which does scheduling, is called?) on osd side. Then I can imagine
that a request to a PG goes to one of the connections by simple
modulo operation (something like PG_id % Number_of_conns). So all
requests to a PG from all clients will be eventually handled by one
of the cpu threads. Something like that?
Yup, basically a set of PGs would get its own crimson-msgr instance
to let clients to talk directly with the proper CPU core – without
crossbar or, in general, any data / message passing between CPU cores
on hot paths.
May I take a look on the link with numbers and what
exactly persistent
object store you've mentioning?
+Mark.
This was the testing Mark has initially mentioned. ceph-osd + BlueStore
has been compared with ceph-osd + MemStore during random reads.
I can't find the spreadsheet but I asked Mark today.
Regards,
Radek
3:37 a.m.
On 1/14/20 2:05 PM, Radoslaw Zarzynski wrote:
On Tue, Jan 14, 2020 at 12:13 PM Roman Penyaev
<rpenyaev(a)suse.de> wrote:
Found it:
https://docs.google.com/spreadsheets/d/1kfzbvtdhUvrzjn9eW0r6Fqrm8gE6X0bmZnE…
Mark
Am I right that you are talking about several
connections between
a primary osd and a single client instance? At least I'm picturing
that each connection represents a software cpu (or how this thread,
which does scheduling, is called?) on osd side. Then I can imagine
that a request to a PG goes to one of the connections by simple
modulo operation (something like PG_id % Number_of_conns). So all
requests to a PG from all clients will be eventually handled by one
of the cpu threads. Something like that?
Yup, basically a set of PGs would get
its own crimson-msgr instance
to let clients to talk directly with the proper CPU core – without
crossbar or, in general, any data / message passing between CPU cores
on hot paths.
May I take a look on the link with numbers and
what exactly persistent
object store you've mentioning?
+Mark.
This was the testing Mark has initially mentioned. ceph-osd + BlueStore
has been compared with ceph-osd + MemStore during random reads.
I can't find the spreadsheet but I asked Mark today.
4:35 p.m.
On 2020-01-14 23:07, Mark Nelson wrote:
On 1/14/20 2:05 PM, Radoslaw Zarzynski wrote:
Thanks for sharing. Do I understand correctly that this ~270k cycles/io
for writes can be treated as a best estimation for objectstore?
Kind of ideal boundary to which we should strive for? (doing comparison
on the same hardware, of course). Since memstore is log-less, this
estimation, of course, is hardly reachable, but can be treated as
a perfect unattainable reference.
--
Roman
On Tue, Jan 14, 2020 at 12:13 PM Roman Penyaev
<rpenyaev(a)suse.de>
wrote:
Found it:
https://docs.google.com/spreadsheets/d/1kfzbvtdhUvrzjn9eW0r6Fqrm8gE6X0bmZnE…
Am I right that you are talking about several
connections between
a primary osd and a single client instance? At least I'm picturing
that each connection represents a software cpu (or how this thread,
which does scheduling, is called?) on osd side. Then I can imagine
that a request to a PG goes to one of the connections by simple
modulo operation (something like PG_id % Number_of_conns). So all
requests to a PG from all clients will be eventually handled by one
of the cpu threads. Something like that?
Yup, basically a set of PGs would get
its own crimson-msgr instance
to let clients to talk directly with the proper CPU core – without
crossbar or, in general, any data / message passing between CPU cores
on hot paths.
May I take a look on the link with numbers and
what exactly
persistent
object store you've mentioning?
+Mark.
This was the testing Mark has initially mentioned. ceph-osd +
BlueStore
has been compared with ceph-osd + MemStore during random reads.
I can't find the spreadsheet but I asked Mark today.
9 p.m.
Hello Roman,
I consider *the difference* between BlueStore and MemStore in ceph-osd
as rough boundary on how much is achievable for this single component.
It's rather unlikely that SeaStore can be beat MemStore. ;-)
Still, it's very, very rough due to the dependencies between components.
For example, it's not impossible that a syscall in messenger potentially
decreases IPC also in ObjectStore. So, please take it with a grain of salt.
Regards,
Radek
On Wed, Jan 15, 2020 at 12:05 PM Roman Penyaev <rpenyaev(a)suse.de> wrote:
On 2020-01-14 23:07, Mark Nelson wrote:
On 1/14/20 2:05 PM, Radoslaw Zarzynski wrote:
Thanks for sharing. Do I understand correctly that this ~270k cycles/io
for writes can be treated as a best estimation for objectstore?
Kind of ideal boundary to which we should strive for? (doing comparison
on the same hardware, of course). Since memstore is log-less, this
estimation, of course, is hardly reachable, but can be treated as
a perfect unattainable reference.
--
Roman
On Tue, Jan 14, 2020 at 12:13 PM Roman Penyaev
<rpenyaev(a)suse.de>
wrote:
Found it:
https://docs.google.com/spreadsheets/d/1kfzbvtdhUvrzjn9eW0r6Fqrm8gE6X0bmZnE…
Am I right that you are talking about several
connections between
a primary osd and a single client instance? At least I'm picturing
that each connection represents a software cpu (or how this thread,
which does scheduling, is called?) on osd side. Then I can imagine
that a request to a PG goes to one of the connections by simple
modulo operation (something like PG_id % Number_of_conns). So all
requests to a PG from all clients will be eventually handled by one
of the cpu threads. Something like that?
Yup, basically a set of PGs would get
its own crimson-msgr instance
to let clients to talk directly with the proper CPU core – without
crossbar or, in general, any data / message passing between CPU cores
on hot paths.
May I take a look on the link with numbers and
what exactly
persistent
object store you've mentioning?
+Mark.
This was the testing Mark has initially mentioned. ceph-osd +
BlueStore
has been compared with ceph-osd + MemStore during random reads.
I can't find the spreadsheet but I asked Mark today.
12 Jan
12 Jan
9:16 p.m.
On 10/01/2020 21.54, Radoslaw Zarzynski wrote:
Hi Roman,
On Thu, Jan 9, 2020 at 2:51 PM Roman Penyaev <rpenyaev(a)suse.de> wrote:
The goal is not to require different optimization techniques or code
paths for the two stacks. And it is a challenge to do this well.
I guess we need a way to provide the state machine to
seastar::input_stream so that it can provide the expected data layout to
data_source. Maybe we need to bypass input_stream completely - it tries
to buffer data by itself but here the application knows better. So
perhaps we need to interact with data_source instead. The application
would provide the buffers for the data_source, and seastar would take
care of the copying if the data_source cannot use the buffer directly
(the native stack case).
> Moreover, crimson-osd imposes on Seastar additional memcpy to
> reconcile those tiny chunks into a flat buffer.
>
> Here is a more detailed gist:
> https://gist.github.com/rzarzynski/a1d67dc39b0ef4d49cb522179b1f3c89.
>
> There are branches (for both Seastar and crimson) with PoC for
> the "input buffer factory" that targets those issues. Performance
> comparison is here:
> https://gist.github.com/rzarzynski/ad0aaa80b26603bc1a803ce0d209ac87.
>
> Also, when narrowing the comparison to async-msgr vs crimson-msgr
> (with ibf) I wouldn't expect too much of a difference. In read tests we're
> observing pretty similar IPC for both crimson-osd and msgr-worker-n
> (single thread profiling). The thing that might change a lot is the native
> stack. In Intel's testing it significantly (up to 30-40% IIRC) improved
> IPC of crimson-msgr. Glued with good SPDK support in Seastore
> it might draw the POSIX stack (and thus the need for ibf) a bit obsolete.
>
> Quick note on the saturation: please be careful when judging it with
> top, pidstat or even perf stat. In contrast to the legacy OSD, Seastar
> does busy-wait for awhile. This greatly exaggerates the CPU utilisation
> for modest workloads. And yes, **crimson is all about the computational
> efficiency**. We're much more interested in cycles/op than in raw IOPS,
> to be honest.
>
> Regards,
> Radek
> _______________________________________________
> Dev mailing list -- dev(a)ceph.io
> To unsubscribe send an email to dev-leave(a)ceph.io
First thing that catches my eye is that for small
blocks there is no big
difference at all, but as the block increases, crimsons iops starts to
decline. Can it be the transport issue? Can be tested as well.
This is a known
issue with the Seastar's POSIX-based network
stack. As Kefu pointed out, even large payloads are retrieved from
kernel with multiple small, fixed-size chunks. That's a matter of how
the internal interfaces were shaped. My personal impression is their
design favors the native stack / DPDK while avoiding differentiated
behavior among the stacks (likely to not surprise developers).
14 Jan
14 Jan
1:26 a.m.
Hi Avi,
On Sun, Jan 12, 2020 at 4:46 PM Avi Kivity <avi(a)scylladb.com> wrote:
The goal is not to require different optimization
techniques or code
paths for the two stacks. And it is a challenge to do this well.
Yeah, it brings even more headache after including a requirement
to handle both stacks efficiently – without sacrificing one for
the sake of another.
I fully understand the strong desire to keep the paths consistent.
It saves a lot of problems that could come because of developing
on POSIX while deploying on native, to exemplify. What I would
likely advocate for is the presence of a switch that would allow
application to differentiate paths for the sake of maximal
performance but under strict rule: if somebody wishes to use
a knife instead of a peeler, it's his own responsibility and problem
if he cuts his fingers off.
The application
would provide the buffers for the data_source, and seastar would take
care of the copying if the data_source cannot use the buffer directly
(the native stack case).
Yes, that's a possibility. The trade-off is introduction of memcpy()
to the native. At the moment I can't say how much a problem it
would be in practice. There was a promising experiment with
crimson-msgr on top of DPDK but the storage component was
excluded (no SPDK at the moment).
Regards,
Radek
3 p.m.
On 13/01/2020 21.56, Radoslaw Zarzynski wrote:
That memcpy would be incurred only if the state machine specified it
needed its own buffer. If it specified it can run from a stack-provided
buffer, it would still be zero copy.
What your proposal does is reuse the read(2) call's copy to userspace
for the application's purposes - the copy is still there. So the native
stack isn't handicapped by this copy, it happens in both.
> At the moment I can't say how much a problem it
> would be in practice. There was a promising experiment with
> crimson-msgr on top of DPDK but the storage component was
> excluded (no SPDK at the moment).
>
Hi Avi,
On Sun, Jan 12, 2020 at 4:46 PM Avi Kivity <avi(a)scylladb.com> wrote:
Well, that's what I would like to avoid. I'd like a developer to know
that if they are developing with the posix stack, the application would
work and work well with the native stack, not that they have to retest
everything.
The goal is not to require different optimization
techniques or code
paths for the two stacks. And it is a challenge to do this well.
Yeah, it brings
even more headache after including a requirement
to handle both stacks efficiently – without sacrificing one for
the sake of another.
I fully understand the strong desire to keep the paths consistent.
It saves a lot of problems that could come because of developing
on POSIX while deploying on native, to exemplify. What I would
likely advocate for is the presence of a switch that would allow
application to differentiate paths for the sake of maximal
performance but under strict rule: if somebody wishes to use
a knife instead of a peeler, it's his own responsibility and problem
if he cuts his fingers off. The
application
would provide the buffers for the data_source, and seastar would take
care of the copying if the data_source cannot use the buffer directly
(the native stack case).
Yes, that's a possibility. The trade-off is
introduction of memcpy()
to the native.
4:46 p.m.
On 14/01/2020 11.30, Avi Kivity wrote:
On 13/01/2020 21.56, Radoslaw Zarzynski wrote:
That memcpy would be incurred only if the state machine specified it
needed its own buffer. If it specified it can run from a
stack-provided buffer, it would still be zero copy.
What your proposal does is reuse the read(2) call's copy to userspace
for the application's purposes - the copy is still there. So the
native stack isn't handicapped by this copy, it happens in both.
Perhaps we can add
placing_data_source data_source::to_placing_data_source() &&
Which converts a regular data_source to a placing_data_source, which
gets as input from the user the buffers to use.
The default placing_data_source would just copy data from the original
data_source to the buffers provided by the user.
The placing_data_source provided by
posix_data_source_impl::to_placing_data_source() would cooperate with
the posix stack to read directly into the buffers provided by the user
(reusing the copy performed by the system call).
If a data movement engine is available, the native stack might program
it to perform the copy.
Hi Avi,
On Sun, Jan 12, 2020 at 4:46 PM Avi Kivity <avi(a)scylladb.com> wrote:
Well, that's what I would like to avoid. I'd like a developer to know
that if they are developing with the posix stack, the application
would work and work well with the native stack, not that they have to
retest everything.
The goal is not to require different optimization
techniques or code
paths for the two stacks. And it is a challenge to do this well.
Yeah, it brings
even more headache after including a requirement
to handle both stacks efficiently – without sacrificing one for
the sake of another.
I fully understand the strong desire to keep the paths consistent.
It saves a lot of problems that could come because of developing
on POSIX while deploying on native, to exemplify. What I would
likely advocate for is the presence of a switch that would allow
application to differentiate paths for the sake of maximal
performance but under strict rule: if somebody wishes to use
a knife instead of a peeler, it's his own responsibility and problem
if he cuts his fingers off. The
application
would provide the buffers for the data_source, and seastar would take
care of the copying if the data_source cannot use the buffer directly
(the native stack case).
Yes, that's a possibility. The trade-off is
introduction of memcpy()
to the native.
15 Jan
15 Jan
1:32 a.m.
Hi Avi,
I responded inline to both of your messages.
On 13/01/2020 21.56, Radoslaw Zarzynski wrote:
Well, that's what I would like to avoid. I'd like a developer to know
that if they are developing with the posix stack, the application would
work and work well with the native stack, not that they have to retest
everything.
I see your point. It's valid one.
That memcpy would be incurred only if the state
machine specified it
needed its own buffer. If it specified it can run from a stack-provided
buffer, it would still be zero copy.
If we could make this decision at run-time, then fine. I'm afraid
application has currently no way to determine which stack is to
be used and adjust the choice dynamically. If so, supporting both
POSIX and native efficiently (with no trade-offs) would boil down
into a compile-time decision, and thus separated builds.
Still, twice the testing. :-(
What your proposal does is reuse the read(2)
call's copy to userspace
for the application's purposes - the copy is still there. So the native
stack isn't handicapped by this copy, it happens in both.
This assumes that crimson will always need to memcpy() the data
retrieved from a network stack. I believe that's not the case.
For the kernel drivers (POSIX stack + kernel's storage)
the read() in Seastar is actually reused to provide kernel with
an opportunity to remap pages, instead of doing memcpy(),
when the retrieved payload is written to storage. This could
happen as the buffer Seastar read into had been properly
aligned.
Apart from the read() itself there is no inherent memcpy() on
the data path. When flowing through it, the payload is always
conveyed as ref-counted scatter-gather list. This stays even
if this SGL has only single segment like when using POSIX
with the particular implementation of input_buffer_factory.
For the user-space drivers (native stack + SPDK) there is a chance
to squeeze memcpy() / remappings entirely. Of course, this
assumes that storage HW is able to deal with inflated SGLs
efficiently. I hope vendors could throw more light on that.
From our last discussion I recall your point about the impact
on cache density (in the meaning of e.g. BlueStore's cache)
ref-counting can impose. For sure segments of our SGL will be
bigger than the actual payload and contain metadata or other
junks.
My answer would be that application's caching policy shouldn't
belong to network layer. It has too little information to judge
whether a given buffer needs to be cached or not. There are
many cases when you won't cache. To exemplify while staying
in the BlueStore's domain: `bluestore_default_buffered_write`
is `false` by default. That is, BlueStore **doesn't cache on writes**.
On Tue, Jan 14, 2020 at 12:16 PM Avi Kivity <avi(a)scylladb.com> wrote:
The default placing_data_source would just copy data
from the original
data_source to the buffers provided by the user.
The placing_data_source provided by
posix_data_source_impl::to_placing_data_source() would cooperate with
the posix stack to read directly into the buffers provided by the user
(reusing the copy performed by the system call).
If a data movement engine is available, the native stack might program
it to perform the copy.
Well, this is about off-loading the memcpy() we would introduce
to the native stack. I still think the best – from the performance's
point-of-view – is to avoid this shuffling at all. The sacrifice for
that would the maintainability concern coming from differentiated
paths in the network layer.
Regards,
Radek
4:52 p.m.
On 14/01/2020 22.02, Radoslaw Zarzynski wrote:
This is the memcpy I was referring to. And the solution I'd like to see
is one where the application tells seastar what buffers it wants to see
the data in, which allows seastar to either direct read() to copy into
those buffers (using the copy it does anyway) or perform the copy itself
(for the native stack case).
Hi Avi,
I responded inline to both of your messages.
I don't understand why you say this. If Seastar performs the memcpy
transparently when needed, why do you need separate builds?
On 13/01/2020 21.56, Radoslaw Zarzynski wrote:
Well, that's what I would like to avoid. I'd like a developer to know
that if they are developing with the posix stack, the application would
work and work well with the native stack, not that they have to retest
everything.
I see your point. It's valid one.
That memcpy would be incurred only if the state
machine specified it
needed its own buffer. If it specified it can run from a stack-provided
buffer, it would still be zero copy.
If we could make this decision at run-time,
then fine. I'm afraid
application has currently no way to determine which stack is to
be used and adjust the choice dynamically. If so, supporting both
POSIX and native efficiently (with no trade-offs) would boil down
into a compile-time decision, and thus separated builds.
Still, twice the testing. :-( What your proposal does is reuse the read(2)
call's copy to userspace
for the application's purposes - the copy is still there. So the native
stack isn't handicapped by this copy, it happens in both.
This assumes that
crimson will always need to memcpy() the data
retrieved from a network stack. I believe that's not the case.
For the kernel drivers (POSIX stack + kernel's storage)
the read() in Seastar is actually reused to provide kernel with
an opportunity to remap pages, instead of doing memcpy(),
when the retrieved payload is written to storage. This could
happen as the buffer Seastar read into had been properly
aligned.
Apart from the read() itself there is no inherent memcpy() on the data path. When flowing through it, the payload is
always
conveyed as ref-counted scatter-gather list. This stays even
if this SGL has only single segment like when using POSIX
with the particular implementation of input_buffer_factory.
For the user-space drivers (native stack + SPDK) there is a chance
to squeeze memcpy() / remappings entirely. Of course, this
assumes that storage HW is able to deal with inflated SGLs
efficiently. I hope vendors could throw more light on that.
If the storage indicates it doesn't need alignment, then the application
avoids telling seastar to read into the application's buffers and
instead accepts the current stack-provided temporary_buffers.
From our last discussion I recall your point about the impact
on cache density (in the meaning of e.g. BlueStore's cache)
ref-counting can impose. For sure segments of our SGL will be
bigger than the actual payload and contain metadata or other
junks.
My answer would be that application's caching policy shouldn't
belong to network layer. It has too little information to judge
whether a given buffer needs to be cached or not. There are
many cases when you won't cache. To exemplify while staying
in the BlueStore's domain: `bluestore_default_buffered_write`
is `false` by default. That is, BlueStore **doesn't cache on writes**.
If you can decide in advance you need to cache, provide the
cache-friendly buffers to seastar and it will make sure the data lands
there (either through the kernel's read() or through its own memcpy).
If you can't make that decision in advance, and you also don't need
alignment for other reasons, then accept seastar's non-aligned buffers
and perform the adjustment yourself if it is later needed.
Is there a case I missed?
On Tue, Jan 14, 2020 at 12:16 PM Avi Kivity <avi(a)scylladb.com> wrote:
If you know you don't need the memcpy, don't provide your pre-allocated
buffers and it won't happen.
The default placing_data_source would just copy
data from the original
data_source to the buffers provided by the user.
The placing_data_source provided by
posix_data_source_impl::to_placing_data_source() would cooperate with
the posix stack to read directly into the buffers provided by the user
(reusing the copy performed by the system call).
If a data movement engine is available, the native stack might program
it to perform the copy.
Well, this is about off-loading the memcpy() we would
introduce
to the native stack. I still think the best – from the performance's
point-of-view – is to avoid this shuffling at all. The sacrifice for
that would the maintainability concern coming from differentiated
paths in the network layer.
6:51 p.m.
Hi Avi,
On Wed, Jan 15, 2020 at 12:22 PM Avi Kivity <avi(a)scylladb.com> wrote:
I don't understand why you say this. If Seastar
performs the memcpy
transparently when needed, why do you need separate builds?
Because I don't want to just handle both stacks. I want to handle them
as efficiently as possible. Preferably, I want to have single build only.
The most efficient strategy for buffer allocation varies depending
on which stacks has been selected:
* for POSIX the best one is to reuse the read() memcpy by providing
application-allocated buffer with proper alignment. The `memcpy()`
already happens even for the sake of preserving the kernel-user
isolation and we can do little about that.
Let's call this strategy "application-provided".
* for native all you want is to grab and ref-count++ the memory buffer
where NIC had thrown the payload. There is no memcpy / page
remapping anywhere. There is no memcpy that could be reused for
anything. Even transparent memcpy is wasteful.
In my understanding which stack is to be used isn't exposed to
application by Seastar yet. If so, this would translate into choosing
the strategy **blindly**. If crimson had decided to go with:
* the native-friendly "stack-provided" while the actual stack is
the POSIX one, then application would have lost the way to
reuse the memcpy;
* the POSIX-friendly "application-provided" but the actual stack
is native – the transparent memcpy() would have happened.
This is the memcpy I was referring to. And the
solution I'd like to see
is one where the application tells seastar what buffers it wants to see
the data in.
I think the misunderstanding comes from our imaginations on what
application tells Seastar. I perceive that your point assumes the choice
is constant:
* ALWAYS use "application-provided" OR
* ALWAYS use "stack-provided".
Unfortunately, underlying OS / hardware characteristics make
this simple approach inefficient. To avoid unnecessary overhead
in both cases application should be able to say:
* WHEN "posix" use "application-provided" AND
* WHEN "native" use "stack-provided".
If you know you don't need the memcpy, don't
provide your pre-allocated
buffers and it won't happen.
This is exactly what I'm targeting. If I could make this decision
(whether to provide an application-allocated buffer or accept
a stack-provided one) conditionally (isNative() / isPosix()), then
it's done! :-)
Regards,
Radek
7:04 p.m.
On 15/01/2020 15.21, Radoslaw Zarzynski wrote:
This is what I don't understand. The application has no requirement to
do something depending on the stack. It has requirements (or
non-requirements) on alignment.
What I propose it:
* WHEN you have a requirement for aligned buffers, use
"application-provided"
* WHEN you do not have a requirement for aligned buffers, use
"stack-provided"
After the applications starts, do you not know whether you have a
requirement for alignment or not?
Why is it dependent on the stack? It should depend only on whether you
need alignment (or more generically, placement) or not.
Let's look at a similar problem and similar solution. Disks either have
a volatile or non-volatile write cache. Applications don't ask whether
the disk has a volatile write cache or not. Instead, they indicate their
requirements. If they want to hit the media they open the file with
O_DSYNC (other methods are available, let's look at this example). The
storage stack then takes care of the rest. If the write cache is not
volatile, then it is enough to sent the write to the disk. If the write
cache is volatile, then the storage stack can set the FUA bit, or if the
write is split, it can issue several volatile writes and then instruct
the disk to flush the write cache, and only then return to the user.
The application indicates to the stack what it requires, rather than
asking the stack how it is implemented. I would like something similar
for data placement.
>
> Regards,
> Radek
>
Hi Avi,
On Wed, Jan 15, 2020 at 12:22 PM Avi Kivity <avi(a)scylladb.com> wrote:
This is what Seastar provides today and I agree it can should be improved.
I don't understand why you say this. If
Seastar performs the memcpy
transparently when needed, why do you need separate builds?
Because I don't
want to just handle both stacks. I want to handle them
as efficiently as possible. Preferably, I want to have single build only.
The most efficient strategy for buffer allocation varies depending
on which stacks has been selected:
* for POSIX the best one is to reuse the read() memcpy by providing
application-allocated buffer with proper alignment. The `memcpy()`
already happens even for the sake of preserving the kernel-user
isolation and we can do little about that.
Let's call this strategy "application-provided".
* for native all you want is to grab and ref-count++ the memory buffer
where NIC had thrown the payload. There is no memcpy / page
remapping anywhere. There is no memcpy that could be reused for
anything. Even transparent memcpy is wasteful.
In my understanding which stack is to be used isn't exposed to
application by Seastar yet. If so, this would translate into choosing
the strategy **blindly**. If crimson had decided to go with:
* the native-friendly "stack-provided" while the actual stack is
the POSIX one, then application would have lost the way to
reuse the memcpy;
* the POSIX-friendly "application-provided" but the actual stack
is native – the transparent memcpy() would have happened. This is the
memcpy I was referring to. And the solution I'd like to see
is one where the application tells seastar what buffers it wants to see
the data in.
I think the misunderstanding comes from our imaginations on what
application tells Seastar. I perceive that your point assumes the choice
is constant:
* ALWAYS use "application-provided" OR
* ALWAYS use "stack-provided".
Unfortunately, underlying OS / hardware characteristics make
this simple approach inefficient. To avoid unnecessary overhead
in both cases application should be able to say:
* WHEN "posix" use "application-provided" AND
* WHEN "native" use "stack-provided". If you know you don't need the memcpy,
don't provide your pre-allocated
buffers and it won't happen.
This is exactly what I'm targeting. If I
could make this decision
(whether to provide an application-allocated buffer or accept
a stack-provided one) conditionally (isNative() / isPosix()), then
it's done! :-)
7:42 p.m.
On Wed, Jan 15, 2020 at 2:34 PM Avi Kivity <avi(a)scylladb.com> wrote:
This is what Seastar provides today and I agree it can
should be improved.
I agree. Let's continue our discussion and try to find the way. :-)
What I propose it:
* WHEN you have a requirement for aligned buffers, use
"application-provided"
* WHEN you do not have a requirement for aligned buffers, use
"stack-provided"
After the applications starts, do you not know whether you have a
requirement for alignment or not?
We have the knowledge on alignment, so let's experiment with
the proposed ruleset to judge performance repercussions.
Today, when crimson-osd is all about the cyan store (simple,
RAM-backed store for testing), we can definitely say there is
no requirement for alignment. Basing on that and the rule:
* WHEN you do not have a requirement for aligned buffers, use
"stack-provided".
Therefore we should opt for "stack-provided". Let's verify
the result:
* if the actual stack is native, everything is OK. There will
be no even single memcpy, no syscall.
* if the actual stack is POSIX, as there is no provided buffer,
there is also no buffer.length. The stack needs to guess
how many bytes read() from the socket. If the guessed
number is too small, the application is hurt by excessive
syscalls. This happens today. :-(
Regards,
Radek
9:54 p.m.
On 15/01/2020 16.12, Radoslaw Zarzynski wrote:
On Wed, Jan 15, 2020 at 2:34 PM Avi Kivity
<avi(a)scylladb.com> wrote:
Ok, so it's not just about alignment, but also about sizes. We can also
allow the application to specify how many bytes it wants to read (in
fact, it can already do that with read_exactly, but input_stream does
not pass the information along).
Let's list the possible cases:
- the application knows nothing (common when parsing a complex stream
containing small objects). This is where Scylla is, similar to an HTTP
server.
- the protocol has rigid structure (fixed size header + variable
payload). The application wants the header in a linearized buffer and
the payload in a free-form iovec. This corresponds to cyanstore.
- the protocol has rigid structure as above. The application wants the
header in a linearized buffer and the payload in its own buffers due to
alignment or ownership requirements. This corresponds to a production
storage server that has alignment requirements for talking to storage
and ownership/placement requirements for caching blocks.
In the first case, input_stream should provide buffers as it reads them.
Buffers can end due to packet boundaries (native stack), input_stream
buffer boundaries (posix stack) or due to exhausting all received data
(both).
In the second case, the socket (perhaps not input_stream) should
linearize the header, provide the payload as a sequence of buffers, and
should attempt not to over-read (over-reading the payload can require
linearization of the next header or trailer)
In the third case, the socket linearizes both the header and payload,
the first into a buffer it allocates by itself, the second into a buffer
provided by the user.
Is this a good set of capabilities to provide?
If it is, then we can implement "linearizes" differently for each stack,
and also depending on whether the buffer is provided by the user or the
stack.
For buffers provided by the stack (which there is only a linearization
requirement, not a placement requirement):
- posix allocates a buffer and issues read() syscalls until the buffer
is full
- native will attempt to temporary_buffer::share() the buffer if it
fits into a packet, and allocate and copy if it does not
For buffers provided by the user (placement requirement)
- posix issues repeated read() syscalls until the buffer is full
- native will memcpy from raw packets into the buffers
Note: "buffer" here can also be an iovec or equivalent. In that case it
will be read into using readv(), and "linearization" only happens within
individual elements of the iovec.
This is what Seastar provides today and I agree
it can should be improved.
I agree. Let's continue our discussion and try to
find the way. :-)
What I propose it:
* WHEN you have a requirement for aligned buffers, use
"application-provided"
* WHEN you do not have a requirement for aligned buffers, use
"stack-provided"
After the applications starts, do you not know whether you have a
requirement for alignment or not?
We have the knowledge on alignment, so let's
experiment with
the proposed ruleset to judge performance repercussions.
Today, when crimson-osd is all about the cyan store (simple,
RAM-backed store for testing), we can definitely say there is
no requirement for alignment. Basing on that and the rule:
* WHEN you do not have a requirement for aligned buffers, use
"stack-provided".
Therefore we should opt for "stack-provided". Let's verify
the result:
* if the actual stack is native, everything is OK. There will
be no even single memcpy, no syscall.
* if the actual stack is POSIX, as there is no provided buffer,
there is also no buffer.length. The stack needs to guess
how many bytes read() from the socket. If the guessed
number is too small, the application is hurt by excessive
syscalls. This happens today. :-(
17 Jan
17 Jan
5:06 a.m.
Hi Avi!
I apologize for the late response and want to thank you for the input.
On Wed, Jan 15, 2020 at 5:24 PM Avi Kivity <avi(a)scylladb.com> wrote:
Ok, so it's not just about alignment, but also
about sizes. We can also
allow the application to specify how many bytes it wants to read (in
fact, it can already do that with read_exactly, but input_stream does
not pass the information along).
Hmm, I believe we need to differentiate between the size of single
linearization request (the size_t instance passed to e.g. read_exactly
in order to get e.g. flat header) and the more general size for the buffer-
to-produce-by-stack. The latter would be acquired on the same layer
as application-provided buffers. However, to not impose extensive
memcpy, the "general size" would need to be basically a hint possible
to ignore by the native stack (just like the input_buffer_factory was).
Let's list the possible cases:
- the application knows nothing (common when parsing a complex stream
containing small objects). This is where Scylla is, similar to an HTTP
server.
- the protocol has rigid structure (fixed size header + variable
payload). The application wants the header in a linearized buffer and
the payload in a free-form iovec. This corresponds to cyanstore.
The Ceph's on-write protocol divides payload into a sequence of
segments. Only one segment may have alignment hint. There is
also fixed-size epilogue as the message's tail. Still, those details
don't seem to mess much here.
- the protocol has rigid structure as above. The
application wants the
header in a linearized buffer and the payload in its own buffers due to
alignment or ownership requirements. This corresponds to a production
storage server that has alignment requirements for talking to storage
and ownership/placement requirements for caching blocks.
[..]
Is this a good set of capabilities to provide?
For the POSIX stack case there is an extra performance requirement:
1 syscall per (not-too-big) message *on average*. It's obtainable as
reading of message N's payload can be combined with N + 1's header.
The issue with "current linearization size" vs "buffer-to-produce-by-
stack size" comes from here. Without the ability to ignore the latter
one in native, crimson would basically always need to linearize entire
message to get the best performance from POSIX. However, this
would hurt the native with excessive memcpy. :-(
When the placement requirement comes to play, 1 read() / msg would
translate into aligning-to-the-middle in the application-provided buffer
(already implemented in the crimson's IBF for the coming "sea store").
The alternative can be the readv().
Likely an application might also want to prefetch to amortize syscall
costs for tiny messages. Still, if it could get control over buffer
allocation / size hint, this thing doesn't look impossible.
If it is, then we can implement "linearizes"
differently for each stack,
and also depending on whether the buffer is provided by the user or the
stack.
Yes, this will provide stacks with the memcpy offload possibility.
For buffers provided by the stack (which there is only
a linearization
requirement, not a placement requirement):
- posix allocates a buffer and issues read() syscalls until the buffer
is full
- native will attempt to temporary_buffer::share() the buffer if it
fits into a packet, and allocate and copy if it does not
That's the painful place. For the sake of the native's efficiency,
linearization should be limited only to small fragments. I'm afraid
this turns the linearization sizes into:
* optional things, basically hints (like the input_buffer_factory was) or
* obligatory parameters controllable on the isNative() / isPosix basis.
Regards,
Radek
1562
days inactive
1569
days old
25 comments
5 participants
participants (5)
-
Anthony D'Atri -
Avi Kivity -
Mark Nelson -
Radoslaw Zarzynski -
Roman Penyaev