crimson-osd vs legacy-osd: should the perf difference be already noticeable?
** resending to this list as well, since did not know that ceph-devel@ is for kernel patches only Hi folks, I was curios to read some early performance benchmarks which compare crimson-osd vs legacy-osd, but could not find any. So eventually decided to do my own micro benchmarks in order to test transport together with PG layer, avoiding any storage costs completely (no reason to test memcpy of memstore which is the only available objectstore for crimson). At least recalling all these ad brochures of seastar which should bring performance on another level by doing preemption in userspace, the difference should be already there and visible in numbers. And yes I'm aware that crimson is in development, but if basic functionality is already supported (like write path), then I can squeeze some numbers. For all testing loads I run original rbd.fio, taken from fio/examples/, of course changing only block size. Since this is a micro benchmark I run only 1 osd cluster. [global] ioengine=rbd clientname=admin pool=rbd rbdname=fio_test rw=randwrite #bs=4k [rbd_iodepth32] iodepth=32 -- Part 1, turn MemStore and cyan_store into null block Testing memcpy is not interesting so in order to run any memstore with 'memstore_debug_omit_block_device_write=true' option set and skip all writes I have to do a small tweak in order to start osd, namely I still need to pass small writes and omit big ones which are sent by the client, something as the following: - if (len > 0 && !local_conf()->memstore_debug_omit_block_device_write) { + + if (len > 0 && + (!local_conf()->memstore_debug_omit_block_device_write || + // We still want cluster meta-data to be saved, so pass only small + // writes, expecting user writes will be >= 4k. + len < 4096)) { *** BTW at the bottom you can find the whole patch with all debug modifications made to deliver these numbers. # legacy-osd, MemStore MON=1 MDS=0 OSD=1 MGR=1 ../src/vstart.sh --memstore -n \ -o 'memstore_debug_omit_block_device_write=true' 4k IOPS=42.7k, BW=167MiB/s, Lat=749.18usec 8k IOPS=40.2k, BW=314MiB/s, Lat=795.03usec 16k IOPS=37.6k, BW=588MiB/s, Lat=849.12usec 32k IOPS=32.0k, BW=1000MiB/s, Lat=998.56usec 64k IOPS=25.5k, BW=1594MiB/s, Lat=1253.99usec 128k IOPS=17.5k, BW=2188MiB/s, Lat=1826.54usec 256k IOPS=10.1k, BW=2531MiB/s, Lat=3157.33usec 512k IOPS=5252, BW=2626MiB/s, Lat=6071.37usec 1m IOPS=2656, BW=2656MiB/s, Lat=12029.65usec # crimson-osd, cyan_store MON=1 MDS=0 OSD=1 MGR=1 ../src/vstart.sh --crimson --memstore -n \ -o 'memstore_debug_omit_block_device_write=true' 4k IOPS=40.2k, BW=157MiB/s, Lat=796.07usec 8k IOPS=37.1k, BW=290MiB/s, Lat=861.51usec 16k IOPS=32.9k, BW=514MiB/s, Lat=970.99usec 32k IOPS=26.1k, BW=815MiB/s, Lat=1225.78usec 64k IOPS=21.3k, BW=1333MiB/s, Lat=1498.92usec 128k IOPS=14.4k, BW=1795MiB/s, Lat=2227.07usec 256k IOPS=6143, BW=1536MiB/s, Lat=5203.70usec 512k IOPS=3776, BW=1888MiB/s, Lat=8464.79usec 1m IOPS=1866, BW=1867MiB/s, Lat=17126.36usec First thing that catches my eye is that for small blocks there is no big difference at all, but as the block increases, crimsons iops starts to decline. Can it be the transport issue? Can be tested as well. -- Part 2, complete writes immediately, even not leaving the transport Would be great to avoid PG logic costs, exactly like we did for objectstore, i.e. the following question can be asked "how fast we can handle writes and complete them immediately from the transport callback and measure socket read/write costs?". I introduced new option 'osd_immediate_completions' and handle it directly from 'OSD::ms_[fast_]dispatch' function replying with success just immediately (for details see patch at the bottom). # legacy-osd MON=1 MDS=0 OSD=1 MGR=1 ../src/vstart.sh --memstore -n \ -o 'osd_immediate_completions=true' 4k IOPS=59.2k, BW=231MiB/s, Lat=539.68usec 8k IOPS=55.1k, BW=430MiB/s, Lat=580.44usec 16k IOPS=50.5k, BW=789MiB/s, Lat=633.03usec 32k IOPS=44.6k, BW=1394MiB/s, Lat=716.74usec 64k IOPS=33.5k, BW=2093MiB/s, Lat=954.60usec 128k IOPS=20.8k, BW=2604MiB/s, Lat=1535.01usec 256k IOPS=10.6k, BW=2642MiB/s, Lat=3026.19usec 512k IOPS=5400, BW=2700MiB/s, Lat=5920.86usec 1m IOPS=2549, BW=2550MiB/s, Lat=12539.40usec # crimson-osd MON=1 MDS=0 OSD=1 MGR=1 ../src/vstart.sh --crimson --memstore -n \ -o 'osd_immediate_completions=true' 4k IOPS=60.2k, BW=235MiB/s, Lat=530.95usec 8k IOPS=52.0k, BW=407MiB/s, Lat=614.21usec 16k IOPS=47.1k, BW=736MiB/s, Lat=678.41usec 32k IOPS=37.8k, BW=1180MiB/s, Lat=846.75usec 64k IOPS=26.6k, BW=1660MiB/s, Lat=1203.51usec 128k IOPS=15.5k, BW=1936MiB/s, Lat=2064.12usec 256k IOPS=7506, BW=1877MiB/s, Lat=4259.19usec 512k IOPS=3941, BW=1971MiB/s, Lat=8112.67usec 1m IOPS=1785, BW=1786MiB/s, Lat=17896.44usec As a summary I can say that for me is quite surprising not to notice any iops improvements on crimson side (not to mention the problem with reading of big blocks). Since I run only 1 osd on one particular load I admit the artificial nature of such tests (thus called micro benchmark), but then on what cluster scale and what benchmark can I run to see some improvements of a new architecture? Roman --- src/common/options.cc | 4 ++++ src/crimson/os/cyan_store.cc | 6 +++++- src/crimson/osd/ops_executer.cc | 4 ++-- src/crimson/osd/osd.cc | 20 ++++++++++++++++++++ src/os/memstore/MemStore.cc | 6 +++++- src/osd/OSD.cc | 24 ++++++++++++++++++++++++ 6 files changed, 60 insertions(+), 4 deletions(-) diff --git a/src/common/options.cc b/src/common/options.cc index d91827c1a803..769666d2955c 100644 --- a/src/common/options.cc +++ b/src/common/options.cc @@ -4234,6 +4234,10 @@ std::vector<Option> get_global_options() { .set_default(false) .set_description(""), + Option("osd_immediate_completions", Option::TYPE_BOOL, Option::LEVEL_ADVANCED) + .set_default(false) + .set_description(""), + // -------------------------- // bluestore diff --git a/src/crimson/os/cyan_store.cc b/src/crimson/os/cyan_store.cc index f0749cb921f9..c05e0e40b721 100644 --- a/src/crimson/os/cyan_store.cc +++ b/src/crimson/os/cyan_store.cc @@ -463,7 +463,11 @@ int CyanStore::_write(const coll_t& cid, const ghobject_t& oid, return -ENOENT; ObjectRef o = c->get_or_create_object(oid); - if (len > 0 && !local_conf()->memstore_debug_omit_block_device_write) { + if (len > 0 && + (!local_conf()->memstore_debug_omit_block_device_write || + // We still want cluster meta-data to be saved, so pass only small + // writes, expecting user writes will be >= 4k. + len < 4096)) { const ssize_t old_size = o->get_size(); o->write(offset, bl); used_bytes += (o->get_size() - old_size); diff --git a/src/crimson/osd/ops_executer.cc b/src/crimson/osd/ops_executer.cc index 13f6f086c4ea..a76fc6e206d8 100644 --- a/src/crimson/osd/ops_executer.cc +++ b/src/crimson/osd/ops_executer.cc @@ -431,8 +431,8 @@ OpsExecuter::execute_osd_op(OSDOp& osd_op) default: logger().warn("unknown op {}", ceph_osd_op_name(op.op)); - throw std::runtime_error( - fmt::format("op '{}' not supported", ceph_osd_op_name(op.op))); + // Without that `fio examples/rbd.fio` hangs on exit + throw ceph::osd::operation_not_supported{}; } } diff --git a/src/crimson/osd/osd.cc b/src/crimson/osd/osd.cc index ddd8742d1a74..737cc266766e 100644 --- a/src/crimson/osd/osd.cc +++ b/src/crimson/osd/osd.cc @@ -17,6 +17,7 @@ #include "messages/MOSDOp.h" #include "messages/MOSDPGLog.h" #include "messages/MOSDRepOpReply.h" +#include "messages/MOSDOpReply.h" #include "messages/MPGStats.h" #include "os/Transaction.h" @@ -881,6 +882,25 @@ seastar::future<> OSD::committed_osd_maps(version_t first, seastar::future<> OSD::handle_osd_op(ceph::net::Connection* conn, Ref<MOSDOp> m) { + + // + // Immediately complete requests even without leaving the transport + // + if (local_conf().get_val<bool>("osd_immediate_completions")) { + m->finish_decode(); + + for (auto op : m->ops) { + if (op.op.op == CEPH_OSD_OP_WRITE && + // Complete big writes only + op.op.extent.length >= 4096) { + + auto reply = make_message<MOSDOpReply>(m.get(), 0, osdmap->get_epoch(), + CEPH_OSD_FLAG_ACK | CEPH_OSD_FLAG_ONDISK, true); + return conn->send(reply); + } + } + } + shard_services.start_operation<ClientRequest>( *this, conn->get_shared(), diff --git a/src/os/memstore/MemStore.cc b/src/os/memstore/MemStore.cc index 05d16edb6cc0..265dc64c808d 100644 --- a/src/os/memstore/MemStore.cc +++ b/src/os/memstore/MemStore.cc @@ -1047,7 +1047,11 @@ int MemStore::_write(const coll_t& cid, const ghobject_t& oid, return -ENOENT; ObjectRef o = c->get_or_create_object(oid); - if (len > 0 && !cct->_conf->memstore_debug_omit_block_device_write) { + if (len > 0 && + (!cct->_conf->memstore_debug_omit_block_device_write || + // We still want cluster meta-data to be saved, so pass only small + // writes, expecting user writes will be bigger than 4k. + len < 4096)) { const ssize_t old_size = o->get_size(); o->write(offset, bl); used_bytes += (o->get_size() - old_size); diff --git a/src/osd/OSD.cc b/src/osd/OSD.cc index 96aed0b706e3..796bf927126f 100644 --- a/src/osd/OSD.cc +++ b/src/osd/OSD.cc @@ -7223,6 +7223,30 @@ void OSD::ms_fast_dispatch(Message *m) return; } + // + // Immediately complete requests even without leaving the transport + // + if (g_conf().get_val<bool>("osd_immediate_completions") && + m->get_type() == CEPH_MSG_OSD_OP) { + MOSDOp *osdop = static_cast<MOSDOp*>(m); + + osdop->finish_decode(); + + for (auto op : osdop->ops) { + if (op.op.op == CEPH_OSD_OP_WRITE && + // Complete big writes only + op.op.extent.length >= 4096) { + MOSDOpReply *reply; + + reply = new MOSDOpReply(osdop, 0, osdmap->get_epoch(), + CEPH_OSD_FLAG_ACK | CEPH_OSD_FLAG_ONDISK, true); + osdop->get_connection()->send_message(reply); + m->put(); + return; + } + } + } + // peering event? switch (m->get_type()) { case CEPH_MSG_PING:
Hi Roman, On Thu, Jan 9, 2020 at 2:51 PM Roman Penyaev <rpenyaev@suse.de> wrote:
First thing that catches my eye is that for small blocks there is no big difference at all, but as the block increases, crimsons iops starts to decline. Can it be the transport issue? Can be tested as well.
This is a known issue with the Seastar's POSIX-based network stack. As Kefu pointed out, even large payloads are retrieved from kernel with multiple small, fixed-size chunks. That's a matter of how the internal interfaces were shaped. My personal impression is their design favors the native stack / DPDK while avoiding differentiated behavior among the stacks (likely to not surprise developers). Moreover, crimson-osd imposes on Seastar additional memcpy to reconcile those tiny chunks into a flat buffer. Here is a more detailed gist: https://gist.github.com/rzarzynski/a1d67dc39b0ef4d49cb522179b1f3c89. There are branches (for both Seastar and crimson) with PoC for the "input buffer factory" that targets those issues. Performance comparison is here: https://gist.github.com/rzarzynski/ad0aaa80b26603bc1a803ce0d209ac87. Also, when narrowing the comparison to async-msgr vs crimson-msgr (with ibf) I wouldn't expect too much of a difference. In read tests we're observing pretty similar IPC for both crimson-osd and msgr-worker-n (single thread profiling). The thing that might change a lot is the native stack. In Intel's testing it significantly (up to 30-40% IIRC) improved IPC of crimson-msgr. Glued with good SPDK support in Seastore it might draw the POSIX stack (and thus the need for ibf) a bit obsolete. Quick note on the saturation: please be careful when judging it with top, pidstat or even perf stat. In contrast to the legacy OSD, Seastar does busy-wait for awhile. This greatly exaggerates the CPU utilisation for modest workloads. And yes, **crimson is all about the computational efficiency**. We're much more interested in cycles/op than in raw IOPS, to be honest. Regards, Radek
Hi Radek, Then taking into account everything you've said what load is a win for crimson? What exactly should I run to notice that all shiny things which seastar provides actually work and bring a value? (and I talk about iops, because that's actually what matters for storage after all). I just want to run a simple benchmark, which can show me a clear difference in iops numbers between legacy and crimson osd. What should I run? -- Roman On 2020-01-10 20:54, Radoslaw Zarzynski wrote:
Hi Roman,
On Thu, Jan 9, 2020 at 2:51 PM Roman Penyaev <rpenyaev@suse.de> wrote:
First thing that catches my eye is that for small blocks there is no big difference at all, but as the block increases, crimsons iops starts to decline. Can it be the transport issue? Can be tested as well.
This is a known issue with the Seastar's POSIX-based network stack. As Kefu pointed out, even large payloads are retrieved from kernel with multiple small, fixed-size chunks. That's a matter of how the internal interfaces were shaped. My personal impression is their design favors the native stack / DPDK while avoiding differentiated behavior among the stacks (likely to not surprise developers). Moreover, crimson-osd imposes on Seastar additional memcpy to reconcile those tiny chunks into a flat buffer.
Here is a more detailed gist: https://gist.github.com/rzarzynski/a1d67dc39b0ef4d49cb522179b1f3c89.
There are branches (for both Seastar and crimson) with PoC for the "input buffer factory" that targets those issues. Performance comparison is here: https://gist.github.com/rzarzynski/ad0aaa80b26603bc1a803ce0d209ac87.
Also, when narrowing the comparison to async-msgr vs crimson-msgr (with ibf) I wouldn't expect too much of a difference. In read tests we're observing pretty similar IPC for both crimson-osd and msgr-worker-n (single thread profiling). The thing that might change a lot is the native stack. In Intel's testing it significantly (up to 30-40% IIRC) improved IPC of crimson-msgr. Glued with good SPDK support in Seastore it might draw the POSIX stack (and thus the need for ibf) a bit obsolete.
Quick note on the saturation: please be careful when judging it with top, pidstat or even perf stat. In contrast to the legacy OSD, Seastar does busy-wait for awhile. This greatly exaggerates the CPU utilisation for modest workloads. And yes, **crimson is all about the computational efficiency**. We're much more interested in cycles/op than in raw IOPS, to be honest.
Regards, Radek
Hi Roman, On Sat, Jan 11, 2020 at 4:49 PM Roman Penyaev <rpenyaev@suse.de> wrote:
Then taking into account everything you've said what load is a win for crimson? What exactly should I run to notice that all shiny things which seastar provides actually work and bring a value?
I usually go with 4 KB random reads using `rados bench`. For the main path It should generate workload very similar to RBD while being much more efficient generator than FIO. The former requires a fair amount of CPUs which can be restricting when doing quick dev tests with multiple instances (crimson requires at least two to saturate) on a HW with limited CPU count.
(and I talk about iops, because that's actually what matters for storage after all).
This requires clarification. IOPS from single OSD instance or from a set of HW? I think that all finally matters is return-on-investment. Number of processes inside a box is an implementation detail.
I just want to run a simple benchmark, which can show me a clear difference in iops numbers between legacy and crimson osd.
Sure. Currently we have the `perf check bot` doing `scripts/run-cbt.sh` on `crimson-osd` to hunt for regressions. However, deploying environment might be a bit time-consuming, so I quickly & roughly compared both OSD implementations in the old school way: https://gist.github.com/rzarzynski/9f92f951c929d4a4ed9a7a13ff156b71 All commands all there. I would be grateful for verifying the results. Regards, Radek
On 2020-01-13 14:36, Radoslaw Zarzynski wrote: [skip]
(and I talk about iops, because that's actually what matters for storage after all).
This requires clarification. IOPS from single OSD instance or from a set of HW? I think that all finally matters is return-on-investment. Number of processes inside a box is an implementation detail.
I do not understand. I talk about simple comparison metric for any storage application - IOPS. Since both storage applications (legacy-osd, crimson-osd) share absolutely the same Ceph spec - that is a fair choice. For example we do not compare replication protocols (strong consistency vs eventual or weak consistency), or client driven replication vs primary copy replication. These parts in our comparison are the same. But what we compare is how fast message is received and reply is sent back. Nothing more. So what else if not IOPS? If I were a game developer I would probably choose fps :)
I just want to run a simple benchmark, which can show me a clear difference in iops numbers between legacy and crimson osd.
Sure. Currently we have the `perf check bot` doing `scripts/run-cbt.sh` on `crimson-osd` to hunt for regressions. However, deploying environment might be a bit time-consuming, so I quickly & roughly compared both OSD implementations in the old school way:
https://gist.github.com/rzarzynski/9f92f951c929d4a4ed9a7a13ff156b71
All commands all there. I would be grateful for verifying the results.
Here is my: https://gist.github.com/rouming/dcdca2d6e23fa7ae9041bbda28efd2a1 legacy-osd: ~130MB/s crimson-osd: ~120MB/s BTW prefill writes are also corresponding to reads on my machine: legacy-osd: 122MB/s crimson-osd: 94MB/s What other loads I can try? -- Roman
Hi Roman, On Mon, Jan 13, 2020 at 5:36 PM Roman Penyaev <rpenyaev@suse.de> wrote:
I do not understand. I talk about simple comparison metric for any storage application - IOPS. Since both storage applications (legacy-osd, crimson-osd) share absolutely the same Ceph spec - that is a fair choice.
That way you're actually thinking about IOPS from an OSD instance *disregarding how much HW resources it spends* to serve your workload. This comparison ignores absolutely fundamental difference in architecture: * crimson-osd is single-threaded at the moment. It won't eat more than 1 CPU core. That's by design. * ceph-osd is multi-threaded. By default single instance has up to 16 `tp_osd_tp` and 3 `msgr-worker-n` threads. This translates into upper, theoretical bound of 19 CPU cores. In practice it's of course much lower but still far above than for crimson-osd. Both implementations share the same restriction: amount invested on hardware resources to run the cluster. How much IOPS you will get from it is determined by the OSD's *computational efficiency*. The goal is to maximize IOPS from fixed set of hardware OR, rephrased, to minimize the hardware resources needed to provide a given amount of IOPS. The problem is awfully similar to the performance-per-watt metric and CPU's power efficiency. Electrical / cooling power is scarce resource just like number of CPU cores is in a Ceph cluster. Regards, Radek
Put another way, consider the savings: o If today you have to deploy dual-socket servers for enough cycles, especially with dmcrypt and during recovery. o Tomorrow you could deploy cost-effective servers with much less expensive single-socket CPUs (and thus freedom from NUMA hassles) alternately: o Denser servers with more OSDs per node but the same number of sockets / cores For larger clusters the CapEx efficiency will be substantial. Maybe a cluster that economics previously limited to HDDs can now be all-SSD. — aad
I do not understand. I talk about simple comparison metric for any storage application - IOPS. Since both storage applications (legacy-osd, crimson-osd) share absolutely the same Ceph spec - that is a fair choice.
That way you're actually thinking about IOPS from an OSD instance *disregarding how much HW resources it spends* to serve your workload. This comparison ignores absolutely fundamental difference in architecture:
* crimson-osd is single-threaded at the moment. It won't eat more than 1 CPU core. That's by design. * ceph-osd is multi-threaded. By default single instance has up to 16 `tp_osd_tp` and 3 `msgr-worker-n` threads. This translates into upper, theoretical bound of 19 CPU cores. In practice it's of course much lower but still far above than for crimson-osd.
Both implementations share the same restriction: amount invested on hardware resources to run the cluster. How much IOPS you will get from it is determined by the OSD's *computational efficiency*. The goal is to maximize IOPS from fixed set of hardware OR, rephrased, to minimize the hardware resources needed to provide a given amount of IOPS.
The problem is awfully similar to the performance-per-watt metric and CPU's power efficiency. Electrical / cooling power is scarce resource just like number of CPU cores is in a Ceph cluster.
Regards, Radek _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On 2020-01-13 19:38, Radoslaw Zarzynski wrote:
Hi Roman,
On Mon, Jan 13, 2020 at 5:36 PM Roman Penyaev <rpenyaev@suse.de> wrote:
I do not understand. I talk about simple comparison metric for any storage application - IOPS. Since both storage applications (legacy-osd, crimson-osd) share absolutely the same Ceph spec - that is a fair choice.
That way you're actually thinking about IOPS from an OSD instance *disregarding how much HW resources it spends* to serve your workload. This comparison ignores absolutely fundamental difference in architecture:
* crimson-osd is single-threaded at the moment. It won't eat more than 1 CPU core. That's by design. * ceph-osd is multi-threaded. By default single instance has up to 16 `tp_osd_tp` and 3 `msgr-worker-n` threads. This translates into upper, theoretical bound of 19 CPU cores. In practice it's of course much lower but still far above than for crimson-osd.
Then probably it makes more sense to execute the same load but with iodepth=1? Otherwise it is not quite fair: legacy-osd is able to execute requests in parallel, but crimson-osd is not. $ bin/rados bench -p test-pool 10 rand -t 1 legacy-osd: 34MB/s crimson-osd: 53MB/s At least this is fair and there is a noticeable difference. Also I'm still curios how fast can be immediate completion of requests, without leaving the transport layer and avoiding PG logic completely. With 'osd_immediate_completions=true' option set (there is a patch in the first email of this thread) the bandwidth is the same for both: $ bin/rados bench -p test-pool 10 write -b 4096 --no-cleanup -t 1 legacy-osd: 63MB/s crimson-osd: 63MB/s Which, I would say, not quite impressive. -- Roman
On Mon, Jan 13, 2020 at 7:38 PM Radoslaw Zarzynski <rzarzyns@redhat.com> wrote:
* crimson-osd is single-threaded at the moment. It won't eat more than 1 CPU core. That's by design.
Quick supplement on that: actually there is a way to let single crimson-osd instance to span multiple CPU cores while preserving the shared-nothing design. Definitely the doors for it should be kept open but – as it requires an extension to the RADOS protocol – it would be preferred to not hurry up. IMHO we should consider it after the seastore (BTW: persistent object store is worth ~200 kcycles / op accordingly to Mark's testing). Regards, Radek
On 2020-01-13 22:11, Radoslaw Zarzynski wrote:
On Mon, Jan 13, 2020 at 7:38 PM Radoslaw Zarzynski <rzarzyns@redhat.com> wrote:
* crimson-osd is single-threaded at the moment. It won't eat more than 1 CPU core. That's by design.
Quick supplement on that: actually there is a way to let single crimson-osd instance to span multiple CPU cores while preserving the shared-nothing design. Definitely the doors for it should be kept open but – as it requires an extension to the RADOS protocol – it would be preferred to not hurry up.
Am I right that you are talking about several connections between a primary osd and a single client instance? At least I'm picturing that each connection represents a software cpu (or how this thread, which does scheduling, is called?) on osd side. Then I can imagine that a request to a PG goes to one of the connections by simple modulo operation (something like PG_id % Number_of_conns). So all requests to a PG from all clients will be eventually handled by one of the cpu threads. Something like that?
IMHO we should consider it after the seastore (BTW: persistent object store is worth ~200 kcycles / op accordingly to Mark's testing).
May I take a look on the link with numbers and what exactly persistent object store you've mentioning? -- Roman
On Tue, Jan 14, 2020 at 12:13 PM Roman Penyaev <rpenyaev@suse.de> wrote:
Am I right that you are talking about several connections between a primary osd and a single client instance? At least I'm picturing that each connection represents a software cpu (or how this thread, which does scheduling, is called?) on osd side. Then I can imagine that a request to a PG goes to one of the connections by simple modulo operation (something like PG_id % Number_of_conns). So all requests to a PG from all clients will be eventually handled by one of the cpu threads. Something like that?
Yup, basically a set of PGs would get its own crimson-msgr instance to let clients to talk directly with the proper CPU core – without crossbar or, in general, any data / message passing between CPU cores on hot paths.
May I take a look on the link with numbers and what exactly persistent object store you've mentioning?
+Mark. This was the testing Mark has initially mentioned. ceph-osd + BlueStore has been compared with ceph-osd + MemStore during random reads. I can't find the spreadsheet but I asked Mark today. Regards, Radek
On 1/14/20 2:05 PM, Radoslaw Zarzynski wrote:
On Tue, Jan 14, 2020 at 12:13 PM Roman Penyaev <rpenyaev@suse.de> wrote:
Am I right that you are talking about several connections between a primary osd and a single client instance? At least I'm picturing that each connection represents a software cpu (or how this thread, which does scheduling, is called?) on osd side. Then I can imagine that a request to a PG goes to one of the connections by simple modulo operation (something like PG_id % Number_of_conns). So all requests to a PG from all clients will be eventually handled by one of the cpu threads. Something like that? Yup, basically a set of PGs would get its own crimson-msgr instance to let clients to talk directly with the proper CPU core – without crossbar or, in general, any data / message passing between CPU cores on hot paths.
May I take a look on the link with numbers and what exactly persistent object store you've mentioning? +Mark. This was the testing Mark has initially mentioned. ceph-osd + BlueStore has been compared with ceph-osd + MemStore during random reads. I can't find the spreadsheet but I asked Mark today.
Found it: https://docs.google.com/spreadsheets/d/1kfzbvtdhUvrzjn9eW0r6Fqrm8gE6X0bmZnEC... Mark
On 2020-01-14 23:07, Mark Nelson wrote:
On 1/14/20 2:05 PM, Radoslaw Zarzynski wrote:
On Tue, Jan 14, 2020 at 12:13 PM Roman Penyaev <rpenyaev@suse.de> wrote:
Am I right that you are talking about several connections between a primary osd and a single client instance? At least I'm picturing that each connection represents a software cpu (or how this thread, which does scheduling, is called?) on osd side. Then I can imagine that a request to a PG goes to one of the connections by simple modulo operation (something like PG_id % Number_of_conns). So all requests to a PG from all clients will be eventually handled by one of the cpu threads. Something like that? Yup, basically a set of PGs would get its own crimson-msgr instance to let clients to talk directly with the proper CPU core – without crossbar or, in general, any data / message passing between CPU cores on hot paths.
May I take a look on the link with numbers and what exactly persistent object store you've mentioning? +Mark. This was the testing Mark has initially mentioned. ceph-osd + BlueStore has been compared with ceph-osd + MemStore during random reads. I can't find the spreadsheet but I asked Mark today.
Found it:
https://docs.google.com/spreadsheets/d/1kfzbvtdhUvrzjn9eW0r6Fqrm8gE6X0bmZnEC...
Thanks for sharing. Do I understand correctly that this ~270k cycles/io for writes can be treated as a best estimation for objectstore? Kind of ideal boundary to which we should strive for? (doing comparison on the same hardware, of course). Since memstore is log-less, this estimation, of course, is hardly reachable, but can be treated as a perfect unattainable reference. -- Roman
Hello Roman, I consider *the difference* between BlueStore and MemStore in ceph-osd as rough boundary on how much is achievable for this single component. It's rather unlikely that SeaStore can be beat MemStore. ;-) Still, it's very, very rough due to the dependencies between components. For example, it's not impossible that a syscall in messenger potentially decreases IPC also in ObjectStore. So, please take it with a grain of salt. Regards, Radek On Wed, Jan 15, 2020 at 12:05 PM Roman Penyaev <rpenyaev@suse.de> wrote:
On 2020-01-14 23:07, Mark Nelson wrote:
On 1/14/20 2:05 PM, Radoslaw Zarzynski wrote:
On Tue, Jan 14, 2020 at 12:13 PM Roman Penyaev <rpenyaev@suse.de> wrote:
Am I right that you are talking about several connections between a primary osd and a single client instance? At least I'm picturing that each connection represents a software cpu (or how this thread, which does scheduling, is called?) on osd side. Then I can imagine that a request to a PG goes to one of the connections by simple modulo operation (something like PG_id % Number_of_conns). So all requests to a PG from all clients will be eventually handled by one of the cpu threads. Something like that? Yup, basically a set of PGs would get its own crimson-msgr instance to let clients to talk directly with the proper CPU core – without crossbar or, in general, any data / message passing between CPU cores on hot paths.
May I take a look on the link with numbers and what exactly persistent object store you've mentioning? +Mark. This was the testing Mark has initially mentioned. ceph-osd + BlueStore has been compared with ceph-osd + MemStore during random reads. I can't find the spreadsheet but I asked Mark today.
Found it:
https://docs.google.com/spreadsheets/d/1kfzbvtdhUvrzjn9eW0r6Fqrm8gE6X0bmZnEC...
Thanks for sharing. Do I understand correctly that this ~270k cycles/io for writes can be treated as a best estimation for objectstore? Kind of ideal boundary to which we should strive for? (doing comparison on the same hardware, of course). Since memstore is log-less, this estimation, of course, is hardly reachable, but can be treated as a perfect unattainable reference.
-- Roman
On 10/01/2020 21.54, Radoslaw Zarzynski wrote:
Hi Roman,
First thing that catches my eye is that for small blocks there is no big difference at all, but as the block increases, crimsons iops starts to decline. Can it be the transport issue? Can be tested as well. This is a known issue with the Seastar's POSIX-based network stack. As Kefu pointed out, even large payloads are retrieved from kernel with multiple small, fixed-size chunks. That's a matter of how
On Thu, Jan 9, 2020 at 2:51 PM Roman Penyaev <rpenyaev@suse.de> wrote: the internal interfaces were shaped. My personal impression is their design favors the native stack / DPDK while avoiding differentiated behavior among the stacks (likely to not surprise developers).
The goal is not to require different optimization techniques or code paths for the two stacks. And it is a challenge to do this well. I guess we need a way to provide the state machine to seastar::input_stream so that it can provide the expected data layout to data_source. Maybe we need to bypass input_stream completely - it tries to buffer data by itself but here the application knows better. So perhaps we need to interact with data_source instead. The application would provide the buffers for the data_source, and seastar would take care of the copying if the data_source cannot use the buffer directly (the native stack case).
Moreover, crimson-osd imposes on Seastar additional memcpy to reconcile those tiny chunks into a flat buffer.
Here is a more detailed gist: https://gist.github.com/rzarzynski/a1d67dc39b0ef4d49cb522179b1f3c89.
There are branches (for both Seastar and crimson) with PoC for the "input buffer factory" that targets those issues. Performance comparison is here: https://gist.github.com/rzarzynski/ad0aaa80b26603bc1a803ce0d209ac87.
Also, when narrowing the comparison to async-msgr vs crimson-msgr (with ibf) I wouldn't expect too much of a difference. In read tests we're observing pretty similar IPC for both crimson-osd and msgr-worker-n (single thread profiling). The thing that might change a lot is the native stack. In Intel's testing it significantly (up to 30-40% IIRC) improved IPC of crimson-msgr. Glued with good SPDK support in Seastore it might draw the POSIX stack (and thus the need for ibf) a bit obsolete.
Quick note on the saturation: please be careful when judging it with top, pidstat or even perf stat. In contrast to the legacy OSD, Seastar does busy-wait for awhile. This greatly exaggerates the CPU utilisation for modest workloads. And yes, **crimson is all about the computational efficiency**. We're much more interested in cycles/op than in raw IOPS, to be honest.
Regards, Radek _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Avi, On Sun, Jan 12, 2020 at 4:46 PM Avi Kivity <avi@scylladb.com> wrote:
The goal is not to require different optimization techniques or code paths for the two stacks. And it is a challenge to do this well.
Yeah, it brings even more headache after including a requirement to handle both stacks efficiently – without sacrificing one for the sake of another. I fully understand the strong desire to keep the paths consistent. It saves a lot of problems that could come because of developing on POSIX while deploying on native, to exemplify. What I would likely advocate for is the presence of a switch that would allow application to differentiate paths for the sake of maximal performance but under strict rule: if somebody wishes to use a knife instead of a peeler, it's his own responsibility and problem if he cuts his fingers off.
The application would provide the buffers for the data_source, and seastar would take care of the copying if the data_source cannot use the buffer directly (the native stack case).
Yes, that's a possibility. The trade-off is introduction of memcpy() to the native. At the moment I can't say how much a problem it would be in practice. There was a promising experiment with crimson-msgr on top of DPDK but the storage component was excluded (no SPDK at the moment). Regards, Radek
On 13/01/2020 21.56, Radoslaw Zarzynski wrote:
Hi Avi,
The goal is not to require different optimization techniques or code paths for the two stacks. And it is a challenge to do this well. Yeah, it brings even more headache after including a requirement to handle both stacks efficiently – without sacrificing one for
On Sun, Jan 12, 2020 at 4:46 PM Avi Kivity <avi@scylladb.com> wrote: the sake of another. I fully understand the strong desire to keep the paths consistent. It saves a lot of problems that could come because of developing on POSIX while deploying on native, to exemplify. What I would likely advocate for is the presence of a switch that would allow application to differentiate paths for the sake of maximal performance but under strict rule: if somebody wishes to use a knife instead of a peeler, it's his own responsibility and problem if he cuts his fingers off.
Well, that's what I would like to avoid. I'd like a developer to know that if they are developing with the posix stack, the application would work and work well with the native stack, not that they have to retest everything.
The application would provide the buffers for the data_source, and seastar would take care of the copying if the data_source cannot use the buffer directly (the native stack case). Yes, that's a possibility. The trade-off is introduction of memcpy() to the native.
That memcpy would be incurred only if the state machine specified it needed its own buffer. If it specified it can run from a stack-provided buffer, it would still be zero copy. What your proposal does is reuse the read(2) call's copy to userspace for the application's purposes - the copy is still there. So the native stack isn't handicapped by this copy, it happens in both.
At the moment I can't say how much a problem it would be in practice. There was a promising experiment with crimson-msgr on top of DPDK but the storage component was excluded (no SPDK at the moment).
On 14/01/2020 11.30, Avi Kivity wrote:
On 13/01/2020 21.56, Radoslaw Zarzynski wrote:
Hi Avi,
The goal is not to require different optimization techniques or code paths for the two stacks. And it is a challenge to do this well. Yeah, it brings even more headache after including a requirement to handle both stacks efficiently – without sacrificing one for
On Sun, Jan 12, 2020 at 4:46 PM Avi Kivity <avi@scylladb.com> wrote: the sake of another. I fully understand the strong desire to keep the paths consistent. It saves a lot of problems that could come because of developing on POSIX while deploying on native, to exemplify. What I would likely advocate for is the presence of a switch that would allow application to differentiate paths for the sake of maximal performance but under strict rule: if somebody wishes to use a knife instead of a peeler, it's his own responsibility and problem if he cuts his fingers off.
Well, that's what I would like to avoid. I'd like a developer to know that if they are developing with the posix stack, the application would work and work well with the native stack, not that they have to retest everything.
The application would provide the buffers for the data_source, and seastar would take care of the copying if the data_source cannot use the buffer directly (the native stack case). Yes, that's a possibility. The trade-off is introduction of memcpy() to the native.
That memcpy would be incurred only if the state machine specified it needed its own buffer. If it specified it can run from a stack-provided buffer, it would still be zero copy.
What your proposal does is reuse the read(2) call's copy to userspace for the application's purposes - the copy is still there. So the native stack isn't handicapped by this copy, it happens in both.
Perhaps we can add placing_data_source data_source::to_placing_data_source() && Which converts a regular data_source to a placing_data_source, which gets as input from the user the buffers to use. The default placing_data_source would just copy data from the original data_source to the buffers provided by the user. The placing_data_source provided by posix_data_source_impl::to_placing_data_source() would cooperate with the posix stack to read directly into the buffers provided by the user (reusing the copy performed by the system call). If a data movement engine is available, the native stack might program it to perform the copy.
Hi Avi, I responded inline to both of your messages.
On 13/01/2020 21.56, Radoslaw Zarzynski wrote: Well, that's what I would like to avoid. I'd like a developer to know that if they are developing with the posix stack, the application would work and work well with the native stack, not that they have to retest everything.
I see your point. It's valid one.
That memcpy would be incurred only if the state machine specified it needed its own buffer. If it specified it can run from a stack-provided buffer, it would still be zero copy.
If we could make this decision at run-time, then fine. I'm afraid application has currently no way to determine which stack is to be used and adjust the choice dynamically. If so, supporting both POSIX and native efficiently (with no trade-offs) would boil down into a compile-time decision, and thus separated builds. Still, twice the testing. :-(
What your proposal does is reuse the read(2) call's copy to userspace for the application's purposes - the copy is still there. So the native stack isn't handicapped by this copy, it happens in both.
This assumes that crimson will always need to memcpy() the data retrieved from a network stack. I believe that's not the case. For the kernel drivers (POSIX stack + kernel's storage) the read() in Seastar is actually reused to provide kernel with an opportunity to remap pages, instead of doing memcpy(), when the retrieved payload is written to storage. This could happen as the buffer Seastar read into had been properly aligned. Apart from the read() itself there is no inherent memcpy() on the data path. When flowing through it, the payload is always conveyed as ref-counted scatter-gather list. This stays even if this SGL has only single segment like when using POSIX with the particular implementation of input_buffer_factory. For the user-space drivers (native stack + SPDK) there is a chance to squeeze memcpy() / remappings entirely. Of course, this assumes that storage HW is able to deal with inflated SGLs efficiently. I hope vendors could throw more light on that. From our last discussion I recall your point about the impact on cache density (in the meaning of e.g. BlueStore's cache) ref-counting can impose. For sure segments of our SGL will be bigger than the actual payload and contain metadata or other junks. My answer would be that application's caching policy shouldn't belong to network layer. It has too little information to judge whether a given buffer needs to be cached or not. There are many cases when you won't cache. To exemplify while staying in the BlueStore's domain: `bluestore_default_buffered_write` is `false` by default. That is, BlueStore **doesn't cache on writes**. On Tue, Jan 14, 2020 at 12:16 PM Avi Kivity <avi@scylladb.com> wrote:
The default placing_data_source would just copy data from the original data_source to the buffers provided by the user.
The placing_data_source provided by posix_data_source_impl::to_placing_data_source() would cooperate with the posix stack to read directly into the buffers provided by the user (reusing the copy performed by the system call).
If a data movement engine is available, the native stack might program it to perform the copy.
Well, this is about off-loading the memcpy() we would introduce to the native stack. I still think the best – from the performance's point-of-view – is to avoid this shuffling at all. The sacrifice for that would the maintainability concern coming from differentiated paths in the network layer. Regards, Radek
On 14/01/2020 22.02, Radoslaw Zarzynski wrote:
Hi Avi,
I responded inline to both of your messages.
On 13/01/2020 21.56, Radoslaw Zarzynski wrote: Well, that's what I would like to avoid. I'd like a developer to know that if they are developing with the posix stack, the application would work and work well with the native stack, not that they have to retest everything. I see your point. It's valid one.
That memcpy would be incurred only if the state machine specified it needed its own buffer. If it specified it can run from a stack-provided buffer, it would still be zero copy. If we could make this decision at run-time, then fine. I'm afraid application has currently no way to determine which stack is to be used and adjust the choice dynamically. If so, supporting both POSIX and native efficiently (with no trade-offs) would boil down into a compile-time decision, and thus separated builds. Still, twice the testing. :-(
I don't understand why you say this. If Seastar performs the memcpy transparently when needed, why do you need separate builds?
What your proposal does is reuse the read(2) call's copy to userspace for the application's purposes - the copy is still there. So the native stack isn't handicapped by this copy, it happens in both. This assumes that crimson will always need to memcpy() the data retrieved from a network stack. I believe that's not the case.
For the kernel drivers (POSIX stack + kernel's storage) the read() in Seastar is actually reused to provide kernel with an opportunity to remap pages, instead of doing memcpy(), when the retrieved payload is written to storage. This could happen as the buffer Seastar read into had been properly aligned.
Apart from the read() itself there is no inherent memcpy() on
This is the memcpy I was referring to. And the solution I'd like to see is one where the application tells seastar what buffers it wants to see the data in, which allows seastar to either direct read() to copy into those buffers (using the copy it does anyway) or perform the copy itself (for the native stack case).
the data path. When flowing through it, the payload is always conveyed as ref-counted scatter-gather list. This stays even if this SGL has only single segment like when using POSIX with the particular implementation of input_buffer_factory.
For the user-space drivers (native stack + SPDK) there is a chance to squeeze memcpy() / remappings entirely. Of course, this assumes that storage HW is able to deal with inflated SGLs efficiently. I hope vendors could throw more light on that.
If the storage indicates it doesn't need alignment, then the application avoids telling seastar to read into the application's buffers and instead accepts the current stack-provided temporary_buffers.
From our last discussion I recall your point about the impact on cache density (in the meaning of e.g. BlueStore's cache) ref-counting can impose. For sure segments of our SGL will be bigger than the actual payload and contain metadata or other junks. My answer would be that application's caching policy shouldn't belong to network layer. It has too little information to judge whether a given buffer needs to be cached or not. There are many cases when you won't cache. To exemplify while staying in the BlueStore's domain: `bluestore_default_buffered_write` is `false` by default. That is, BlueStore **doesn't cache on writes**.
If you can decide in advance you need to cache, provide the cache-friendly buffers to seastar and it will make sure the data lands there (either through the kernel's read() or through its own memcpy). If you can't make that decision in advance, and you also don't need alignment for other reasons, then accept seastar's non-aligned buffers and perform the adjustment yourself if it is later needed. Is there a case I missed?
The default placing_data_source would just copy data from the original data_source to the buffers provided by the user.
The placing_data_source provided by posix_data_source_impl::to_placing_data_source() would cooperate with the posix stack to read directly into the buffers provided by the user (reusing the copy performed by the system call).
If a data movement engine is available, the native stack might program it to perform the copy. Well, this is about off-loading the memcpy() we would introduce to the native stack. I still think the best – from the performance's
On Tue, Jan 14, 2020 at 12:16 PM Avi Kivity <avi@scylladb.com> wrote: point-of-view – is to avoid this shuffling at all. The sacrifice for that would the maintainability concern coming from differentiated paths in the network layer.
If you know you don't need the memcpy, don't provide your pre-allocated buffers and it won't happen.
Hi Avi, On Wed, Jan 15, 2020 at 12:22 PM Avi Kivity <avi@scylladb.com> wrote:
I don't understand why you say this. If Seastar performs the memcpy transparently when needed, why do you need separate builds?
Because I don't want to just handle both stacks. I want to handle them as efficiently as possible. Preferably, I want to have single build only. The most efficient strategy for buffer allocation varies depending on which stacks has been selected: * for POSIX the best one is to reuse the read() memcpy by providing application-allocated buffer with proper alignment. The `memcpy()` already happens even for the sake of preserving the kernel-user isolation and we can do little about that. Let's call this strategy "application-provided". * for native all you want is to grab and ref-count++ the memory buffer where NIC had thrown the payload. There is no memcpy / page remapping anywhere. There is no memcpy that could be reused for anything. Even transparent memcpy is wasteful. In my understanding which stack is to be used isn't exposed to application by Seastar yet. If so, this would translate into choosing the strategy **blindly**. If crimson had decided to go with: * the native-friendly "stack-provided" while the actual stack is the POSIX one, then application would have lost the way to reuse the memcpy; * the POSIX-friendly "application-provided" but the actual stack is native – the transparent memcpy() would have happened.
This is the memcpy I was referring to. And the solution I'd like to see is one where the application tells seastar what buffers it wants to see the data in.
I think the misunderstanding comes from our imaginations on what application tells Seastar. I perceive that your point assumes the choice is constant: * ALWAYS use "application-provided" OR * ALWAYS use "stack-provided". Unfortunately, underlying OS / hardware characteristics make this simple approach inefficient. To avoid unnecessary overhead in both cases application should be able to say: * WHEN "posix" use "application-provided" AND * WHEN "native" use "stack-provided".
If you know you don't need the memcpy, don't provide your pre-allocated buffers and it won't happen.
This is exactly what I'm targeting. If I could make this decision (whether to provide an application-allocated buffer or accept a stack-provided one) conditionally (isNative() / isPosix()), then it's done! :-) Regards, Radek
On 15/01/2020 15.21, Radoslaw Zarzynski wrote:
Hi Avi,
On Wed, Jan 15, 2020 at 12:22 PM Avi Kivity <avi@scylladb.com> wrote:
I don't understand why you say this. If Seastar performs the memcpy transparently when needed, why do you need separate builds? Because I don't want to just handle both stacks. I want to handle them as efficiently as possible. Preferably, I want to have single build only. The most efficient strategy for buffer allocation varies depending on which stacks has been selected:
* for POSIX the best one is to reuse the read() memcpy by providing application-allocated buffer with proper alignment. The `memcpy()` already happens even for the sake of preserving the kernel-user isolation and we can do little about that. Let's call this strategy "application-provided". * for native all you want is to grab and ref-count++ the memory buffer where NIC had thrown the payload. There is no memcpy / page remapping anywhere. There is no memcpy that could be reused for anything. Even transparent memcpy is wasteful.
In my understanding which stack is to be used isn't exposed to application by Seastar yet. If so, this would translate into choosing the strategy **blindly**. If crimson had decided to go with:
* the native-friendly "stack-provided" while the actual stack is the POSIX one, then application would have lost the way to reuse the memcpy; * the POSIX-friendly "application-provided" but the actual stack is native – the transparent memcpy() would have happened.
This is what Seastar provides today and I agree it can should be improved.
This is the memcpy I was referring to. And the solution I'd like to see is one where the application tells seastar what buffers it wants to see the data in. I think the misunderstanding comes from our imaginations on what application tells Seastar. I perceive that your point assumes the choice is constant:
* ALWAYS use "application-provided" OR * ALWAYS use "stack-provided".
Unfortunately, underlying OS / hardware characteristics make this simple approach inefficient. To avoid unnecessary overhead in both cases application should be able to say:
* WHEN "posix" use "application-provided" AND * WHEN "native" use "stack-provided".
This is what I don't understand. The application has no requirement to do something depending on the stack. It has requirements (or non-requirements) on alignment. What I propose it: * WHEN you have a requirement for aligned buffers, use "application-provided" * WHEN you do not have a requirement for aligned buffers, use "stack-provided" After the applications starts, do you not know whether you have a requirement for alignment or not?
If you know you don't need the memcpy, don't provide your pre-allocated buffers and it won't happen. This is exactly what I'm targeting. If I could make this decision (whether to provide an application-allocated buffer or accept a stack-provided one) conditionally (isNative() / isPosix()), then it's done! :-)
Why is it dependent on the stack? It should depend only on whether you need alignment (or more generically, placement) or not. Let's look at a similar problem and similar solution. Disks either have a volatile or non-volatile write cache. Applications don't ask whether the disk has a volatile write cache or not. Instead, they indicate their requirements. If they want to hit the media they open the file with O_DSYNC (other methods are available, let's look at this example). The storage stack then takes care of the rest. If the write cache is not volatile, then it is enough to sent the write to the disk. If the write cache is volatile, then the storage stack can set the FUA bit, or if the write is split, it can issue several volatile writes and then instruct the disk to flush the write cache, and only then return to the user. The application indicates to the stack what it requires, rather than asking the stack how it is implemented. I would like something similar for data placement.
Regards, Radek
On Wed, Jan 15, 2020 at 2:34 PM Avi Kivity <avi@scylladb.com> wrote:
This is what Seastar provides today and I agree it can should be improved.
I agree. Let's continue our discussion and try to find the way. :-)
What I propose it:
* WHEN you have a requirement for aligned buffers, use "application-provided"
* WHEN you do not have a requirement for aligned buffers, use "stack-provided"
After the applications starts, do you not know whether you have a requirement for alignment or not?
We have the knowledge on alignment, so let's experiment with the proposed ruleset to judge performance repercussions. Today, when crimson-osd is all about the cyan store (simple, RAM-backed store for testing), we can definitely say there is no requirement for alignment. Basing on that and the rule: * WHEN you do not have a requirement for aligned buffers, use "stack-provided". Therefore we should opt for "stack-provided". Let's verify the result: * if the actual stack is native, everything is OK. There will be no even single memcpy, no syscall. * if the actual stack is POSIX, as there is no provided buffer, there is also no buffer.length. The stack needs to guess how many bytes read() from the socket. If the guessed number is too small, the application is hurt by excessive syscalls. This happens today. :-( Regards, Radek
On 15/01/2020 16.12, Radoslaw Zarzynski wrote:
On Wed, Jan 15, 2020 at 2:34 PM Avi Kivity <avi@scylladb.com> wrote:
This is what Seastar provides today and I agree it can should be improved. I agree. Let's continue our discussion and try to find the way. :-)
What I propose it:
* WHEN you have a requirement for aligned buffers, use "application-provided"
* WHEN you do not have a requirement for aligned buffers, use "stack-provided"
After the applications starts, do you not know whether you have a requirement for alignment or not? We have the knowledge on alignment, so let's experiment with the proposed ruleset to judge performance repercussions.
Today, when crimson-osd is all about the cyan store (simple, RAM-backed store for testing), we can definitely say there is no requirement for alignment. Basing on that and the rule:
* WHEN you do not have a requirement for aligned buffers, use "stack-provided".
Therefore we should opt for "stack-provided". Let's verify the result:
* if the actual stack is native, everything is OK. There will be no even single memcpy, no syscall. * if the actual stack is POSIX, as there is no provided buffer, there is also no buffer.length. The stack needs to guess how many bytes read() from the socket. If the guessed number is too small, the application is hurt by excessive syscalls. This happens today. :-(
Ok, so it's not just about alignment, but also about sizes. We can also allow the application to specify how many bytes it wants to read (in fact, it can already do that with read_exactly, but input_stream does not pass the information along). Let's list the possible cases: - the application knows nothing (common when parsing a complex stream containing small objects). This is where Scylla is, similar to an HTTP server. - the protocol has rigid structure (fixed size header + variable payload). The application wants the header in a linearized buffer and the payload in a free-form iovec. This corresponds to cyanstore. - the protocol has rigid structure as above. The application wants the header in a linearized buffer and the payload in its own buffers due to alignment or ownership requirements. This corresponds to a production storage server that has alignment requirements for talking to storage and ownership/placement requirements for caching blocks. In the first case, input_stream should provide buffers as it reads them. Buffers can end due to packet boundaries (native stack), input_stream buffer boundaries (posix stack) or due to exhausting all received data (both). In the second case, the socket (perhaps not input_stream) should linearize the header, provide the payload as a sequence of buffers, and should attempt not to over-read (over-reading the payload can require linearization of the next header or trailer) In the third case, the socket linearizes both the header and payload, the first into a buffer it allocates by itself, the second into a buffer provided by the user. Is this a good set of capabilities to provide? If it is, then we can implement "linearizes" differently for each stack, and also depending on whether the buffer is provided by the user or the stack. For buffers provided by the stack (which there is only a linearization requirement, not a placement requirement): - posix allocates a buffer and issues read() syscalls until the buffer is full - native will attempt to temporary_buffer::share() the buffer if it fits into a packet, and allocate and copy if it does not For buffers provided by the user (placement requirement) - posix issues repeated read() syscalls until the buffer is full - native will memcpy from raw packets into the buffers Note: "buffer" here can also be an iovec or equivalent. In that case it will be read into using readv(), and "linearization" only happens within individual elements of the iovec.
Hi Avi! I apologize for the late response and want to thank you for the input. On Wed, Jan 15, 2020 at 5:24 PM Avi Kivity <avi@scylladb.com> wrote:
Ok, so it's not just about alignment, but also about sizes. We can also allow the application to specify how many bytes it wants to read (in fact, it can already do that with read_exactly, but input_stream does not pass the information along).
Hmm, I believe we need to differentiate between the size of single linearization request (the size_t instance passed to e.g. read_exactly in order to get e.g. flat header) and the more general size for the buffer- to-produce-by-stack. The latter would be acquired on the same layer as application-provided buffers. However, to not impose extensive memcpy, the "general size" would need to be basically a hint possible to ignore by the native stack (just like the input_buffer_factory was).
Let's list the possible cases:
- the application knows nothing (common when parsing a complex stream containing small objects). This is where Scylla is, similar to an HTTP server.
- the protocol has rigid structure (fixed size header + variable payload). The application wants the header in a linearized buffer and the payload in a free-form iovec. This corresponds to cyanstore.
The Ceph's on-write protocol divides payload into a sequence of segments. Only one segment may have alignment hint. There is also fixed-size epilogue as the message's tail. Still, those details don't seem to mess much here.
- the protocol has rigid structure as above. The application wants the header in a linearized buffer and the payload in its own buffers due to alignment or ownership requirements. This corresponds to a production storage server that has alignment requirements for talking to storage and ownership/placement requirements for caching blocks.
[..]
Is this a good set of capabilities to provide?
For the POSIX stack case there is an extra performance requirement: 1 syscall per (not-too-big) message *on average*. It's obtainable as reading of message N's payload can be combined with N + 1's header. The issue with "current linearization size" vs "buffer-to-produce-by- stack size" comes from here. Without the ability to ignore the latter one in native, crimson would basically always need to linearize entire message to get the best performance from POSIX. However, this would hurt the native with excessive memcpy. :-( When the placement requirement comes to play, 1 read() / msg would translate into aligning-to-the-middle in the application-provided buffer (already implemented in the crimson's IBF for the coming "sea store"). The alternative can be the readv(). Likely an application might also want to prefetch to amortize syscall costs for tiny messages. Still, if it could get control over buffer allocation / size hint, this thing doesn't look impossible.
If it is, then we can implement "linearizes" differently for each stack, and also depending on whether the buffer is provided by the user or the stack.
Yes, this will provide stacks with the memcpy offload possibility.
For buffers provided by the stack (which there is only a linearization requirement, not a placement requirement):
- posix allocates a buffer and issues read() syscalls until the buffer is full
- native will attempt to temporary_buffer::share() the buffer if it fits into a packet, and allocate and copy if it does not
That's the painful place. For the sake of the native's efficiency, linearization should be limited only to small fragments. I'm afraid this turns the linearization sizes into: * optional things, basically hints (like the input_buffer_factory was) or * obligatory parameters controllable on the isNative() / isPosix basis. Regards, Radek
participants (5)
-
Anthony D'Atri
-
Avi Kivity
-
Mark Nelson
-
Radoslaw Zarzynski
-
Roman Penyaev