On 2020-02-03 13:06, Gregory Farnum wrote:
On Mon, Jan 27, 2020 at 2:58 PM Roman Penyaev <rpenyaev@suse.de> wrote:
rados snapshots are based on clone, right? Clone operation should follow the sync path (through the single primary) but still can be a bit tricky (requires object to be identical on all replicas to the moment, when clone is done) and can be implemented by communicating between replicas, where primary is the main coordinator. Here is what I imagine (some sort of lazy data migration)
Uh, they are clones, but not in the way you're thinking/hoping. Client IO *can* (but does not need to) include a SnapSet, which contains data about the snapshots that logically exist (but may or may not have been seen by any particular object yet). When the client does a write with a snapid the object doesn't already contain, the OSD does a clone locally. And then any subsequent write applies to the new head, not the old snapid. So unfortunately neither the client nor the OSD has the slightest idea whether a particular write operation requires any snapshot work until it arrives at the OSD.
Yes, here what I've missed is the fact, that there is no any special command for doing a local clone, the IO with different snapid itself is the initiator of a clone. But even this does not change the main logic: since each replica receives the same IO with different snap id which should spawns a local clone, it should not be difficult to make this cloned object in sync on all replicas and then reply to the client with IO completion. But this extra logic will be needed only in the case of client-based replication. Let's forget about it. I feel that this is quite a popular proposal for Ceph and people tend to defend exactly this particular case. I still do not refuse the idea to have a client responsible for fanning out the write IOs, but what I want to stress, that it is not that important. It is just the option which I dream to have in my-perfect-storage. But what is important in what I proposed is relaxation of strong ordering.
<snip>
Everything which is not plain read/write is treated as management or metadata requests.
(Among other things, any write op has the potential to change the object’s size and that needs to be ordered with truncate operations.)
Why these two operations have to be ordered? No, I will ask another way. Why distributed storage should care about the order of these two operations? That what I do not understand. Why client can't be responsible for proper waiting of IO and only then issuing a truncate? (direct analogy: you issue an IO to your file and do not wait for a completion, then you truncate your file, what is the result?)
But if we are talking about concurrent clients case, when one of the clients issues write, meanwhile another one issues truncate, than I do not understand how does the sync log help, because the primary replica can receive these two requests at any order (we assume no proper locking is used, right?)
If a client dispatches multiple operations on a single object, we guarantee they are ordered in the same order they were dispatched by the client. So he can do async truncate, async write, async read, async append, whatever from a single thread and we promise to process them in that order.
Yes, exactly. RADOS guarantees that a single thread can throw out many async requests and these requests have exactly the same sequence on each replica as these requests were sent from the thread. This is what imposes bunch of restrictions on major layers: PGLog and ObjectStore. So my question is: what are exactly the cases when rbd or cephfs need to throw bunch of overlapping requests? (under overlapping I mean pairs like: read and write to the same offset, create and delete of the same object, truncate and write, etc). Since I don't see the whole complicated picture I can speculate only using the simple facts: vfs layer in the kernel distinguishes metadata path from data path, so fs has to be sure that metadata is durable on the disk, before any IO comes, thus you have to wait for completion of metadata updates; block layer in its turn does not care about ordering, thus requests ordering and flushes is the responsibility of fs on top. So RBD (since it is a block device) should not care about ordering either. Of course metadata path is different (rbd layering, snapshots, etc), but again why RBD can't wait for a completion of metadata requests? And the crucial thing which I want to understand is would it be possible to change clients (cephfs and rbd) in order to relax ordering and make an explicit wait for metadata requests? The negative answer on this question makes the whole proposal and discussion on this topic senseless, because client-based replication (on which we were concentrated so far) is in my wish list and does not influence (or at least not so much) the whole picture.
Some of that could probably be maintained in the client library rather than on the OSDs, but not all of it given the timestamp-based retries you describe and the problem of snapshots I mentioned above.
Not all of them, that's why those which can not, go through the primary. But I got your point.
Basically what I'm getting at is that given the way the RADOS protocol works, we really don't have any ops which are plain read/write. Especially including some of the ones you care about most -- for instance, RBD volumes without an object map include an IO hint on every write, to make sure that any newly-created objects get set to the proper 4MB size. These IO hints mean they're all compound operations, not single-operation writes!
Seems in RBD you already have two paths: w/ and w/o object map, right? Then why a new path with a raw CEPH_OSD_OP_WRITE can't be a modification which let's IO go straight to the replica and then straight to the disk?
Of course, all these issues are not the result of changing our durability guarantees, but of trying to provide client-side replication...
Ok, let's remove client-side replication from a discussion for a while. RADOS ordering constraints - this is what seems important.
Now, making these changes isn’t necessarily bad if we want to develop a faster but less ridiculously-consistent storage system to better serve the needs of the interfaces that actually get deployed — I have long found it a little weird that RADOS, a strictly-consistent transactional object store, is one of the premier providers of virtual block-device IO and S3 storage. But if that’s the goal, we should embrace being not-RADOS and be willing to explicitly take much larger departures from it than just “in the happy path we drop ordering and fall back to backfill if there’s a problem”.
Current constraints are blockers for the IO performance. It does not matter how much we squeeze from the CPU (crimson project), unless we can't relax IO ordering or reduce journaling effects, the overall CPU cycles improvements can be not so impressive.
so I hope Ceph can make a step forward and be less conservative, especially when we have a hardware, which breaks all the possible rules.
The second big point is that if you want to have a happy path and a fallback ordered path, you’ll need to map out in a lot more detail how those interact and how the clients and OSDs switch between them. Ideas like this have come up before but almost every one (or literally every one?) has had a fatal flaw that prevented it actually being safe.
Here I rely on a fact, that replicas know the PG state (as it is right now). If PG is active and clean then replica accepts IO. If not - IO is rejected with the proper error: "dear client, go to the primary, I'm not in the condition to serve your request, but primary can".
Here several scenarios are possible. Client was the first one who observes a replica in not a healthy state. We can expect all other replicas will observe the same not healthy state sooner, but client can propagate this information to other replicas in PG (need to be discussed in detail).
"Not a healthy state" isn't really meaningful in Ceph — we make decisions based on the settings of the newest OSDMap we have access to, but peer OSDs might or might not match that state. When the primary processes an OSDMap marking one of the peers down and he sets it to degraded, there's a window where the peers haven't seen that update. The clients will probably take even longer.
Since clients are the request initiators they will immediately observe the difference in osdmap version just because one of replicas replies back with osdmap version > than client knows about. Here there is no any difference between client which acts as the primary in client- driven replication and a primary in primary-copy model.
And merely not getting an op reply as fast as the client wants isn't indicative of anything that RADOS cares about. Those states and the transitions between them and the recovery logic for ops-in-flight are all very hard to get right, and having the solutions mapped out in detail is a requirement for merging any kind of change in RADOS.
In a client-driven case I talk about raw CEPH_OSD_OP_WRITE* requests only (others follow the primary path). So recovery for this particular case (OP_WRITE* requests) is block based and does not imply any log-based replication. And yes, this brakes RADOS guarantees: WRITE requests are not ordered in respect to other requests.
<snip>
*Hybrid* client-side replication :) When client is responsible for fanning out write requests only in case of healthy pg.
It is frequently undesirable since OSDs tend to have lower latency and more bandwidth to their peers than the clients do to the OSDs;
Latency is the answer. I want to squeeze everything from RDMA. For current Ceph RDMA is dead. Basically for current implementation any per-client improvements on transport side bring nothing. (I spent some time poking the protocol v1 and had a good speed up on transport side, which is unnoticed for the whole per-client IO performance. sigh)
Can you explain why client-side replication over RDMA is a better idea than over ethernet IP?
Seems there is a confusion introduced by me. I do not want to mix RDMA and client-side replication cases, so let me rephrase my previous answer: if we have a client-driven replication and no ordering on IO then it would be great to have an RDMA transport, because then we would have a microseconds latency from a client to all replicas. If we do not have a client-drive replication, but read/write IO ordering is relaxed and writes can go directly to disk bypassing journaling on objectstore side, then it would be also great to have an RDMA transport, but with one extra network hop, so latency is increased, but ok, I can deal with it. But if we have log-based replication for IO (what we have just right now), strict ordering for the whole PG, full journaling on all the data, then it does not matter what transport to use - transport is not a bottleneck, so of course in this case TCP (UDP) socket perfectly copes with its task. No doubts. So again: I just do not want an extra hop (client-driven replication). And for me latency matters (thus RDMA).
Like I said with math, I think in most cases it is actually slower,
My equations are much simpler :) Having one extra hop on the fast IO path doubles network latency.
and it DEFINITELY makes harder all the other kinds of changes you want to make. I think you will be a lot happier if you drop that.
Let's stop discussing it for a while. RADOS API ordering constraints are of much greater importance to me. So if is possible to modify clients in order to teach them to wait for completions properly and never let overlapping IO happen (not-RADOS anymore), then would be great to continue discussion on that topic. If this is an impossible task - well, then cephfs/rbd are not good candidates and probably other cluster file systems can fit better to cope with more relaxed API.
(Also: we are doing a lot of work where read-from-replica will become desirable for things like rack-local reads and not being able to do that would be sad.)
What is this exactly rack-local reads? Could you please elaborate or give me the link? -- Roman