6 Mar
2020
6 Mar
'20
12:04 p.m.
Hoi,
When writting a Rados/Rbd client I noticed that the sequence:
r = rbd_aio_readv(ri->ri_image, iov, iovcnt, offset, comp);
r = rbd_aio_wait_for_complete(comp);
nbytes = rbd_aio_get_return_value(comp);
returns the number of bytes read in nbytes.
Now if I do the same but with `rbd_aio_write(v)` I only receive
zero as value...
I would have expected that rbd_aio_get_return_value would more
or less function like aio_return(2):
DESCRIPTION
The aio_return() system call returns the final status of the
asynchronous
I/O request associated with the structure pointed to by iocb.
RETURN VALUES
If the asynchronous I/O request has completed, the status is
returned as
described in read(2), write(2), or fsync(2). Otherwise, aio_return()
returns -1 and sets errno to indicate the error condition.
So I was expecting the amount of bytes written?
--WjW
7 Mar
7 Mar
2:13 p.m.
On Fri, Mar 6, 2020 at 8:04 AM Willem Jan Withagen <wjw(a)digiware.nl> wrote:
Hoi,
When writting a Rados/Rbd client I noticed that the sequence:
r = rbd_aio_readv(ri->ri_image, iov, iovcnt, offset, comp);
r = rbd_aio_wait_for_complete(comp);
nbytes = rbd_aio_get_return_value(comp);
returns the number of bytes read in nbytes.
Now if I do the same but with `rbd_aio_write(v)` I only receive
zero as value...
I would have expected that rbd_aio_get_return_value would more
or less function like aio_return(2):
DESCRIPTION
The aio_return() system call returns the final status of the
asynchronous
I/O request associated with the structure pointed to by iocb.
RETURN VALUES
If the asynchronous I/O request has completed, the status is
returned as
described in read(2), write(2), or fsync(2). Otherwise, aio_return()
returns -1 and sets errno to indicate the error condition.
So I was expecting the amount of bytes written?
Understandable, but for "distributed systems are hard" reasons, writes
return either 0 (success) or -ERRNO. [1] I guess librbd could
translate that back into the given write size, but it's never come up
as an issue and might have some weird edge cases?
-Greg
[1]: Returning anything else requires recording the return value as an
additional write, or part of the write, so that we give out the same
answer on replay of the write request. We've been talking for years
about extending the protocol+implementation to allow sending back 32
or 64 bytes of data from a write, and...I can't recall if one of those
times it finally got done.
--WjW
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io
4:39 p.m.
On 7-3-2020 19:13, Gregory Farnum wrote:
On Fri, Mar 6, 2020 at 8:04 AM Willem Jan Withagen
<wjw(a)digiware.nl> wrote:
Hi Greg,
Thanx for the info.
I can count what needs to be written from the elements in iov[]. So that
I can fix
in my code. But if -ERRNO is returned could there be a partial write?
I tried to find something documentation on the client stuff in
librados/rbd, but I
could not find any other than testcode in unittest_librbd. That gives
an idea,
but intricate details are not in there.
--WjW
Hoi,
When writting a Rados/Rbd client I noticed that the sequence:
r = rbd_aio_readv(ri->ri_image, iov, iovcnt, offset, comp);
r = rbd_aio_wait_for_complete(comp);
nbytes = rbd_aio_get_return_value(comp);
returns the number of bytes read in nbytes.
Now if I do the same but with `rbd_aio_write(v)` I only receive
zero as value...
I would have expected that rbd_aio_get_return_value would more
or less function like aio_return(2):
DESCRIPTION
The aio_return() system call returns the final status of the
asynchronous
I/O request associated with the structure pointed to by iocb.
RETURN VALUES
If the asynchronous I/O request has completed, the status is
returned as
described in read(2), write(2), or fsync(2). Otherwise, aio_return()
returns -1 and sets errno to indicate the error condition.
So I was expecting the amount of bytes written?
Understandable, but for
"distributed systems are hard" reasons, writes
return either 0 (success) or -ERRNO. [1] I guess librbd could
translate that back into the given write size, but it's never come up
as an issue and might have some weird edge cases?
-Greg
[1]: Returning anything else requires recording the return value as an
additional write, or part of the write, so that we give out the same
answer on replay of the write request. We've been talking for years
about extending the protocol+implementation to allow sending back 32
or 64 bytes of data from a write, and...I can't recall if one of those
times it finally got done.
4:44 p.m.
On Sat, Mar 7, 2020 at 12:39 PM Willem Jan Withagen <wjw(a)digiware.nl> wrote:
On 7-3-2020 19:13, Gregory Farnum wrote:
If RADOS returns an error code, it won't commit anything.
I'm not sure if librbd can issue single writes which turn into
multiple RADOS writes (by involving multiple objects); I suspect in a
sane configuration it can't but maybe you could coerce it with weird
stripe parameters or block sizes...Jason?
-Greg
On Fri, Mar 6, 2020 at 8:04 AM Willem Jan
Withagen <wjw(a)digiware.nl> wrote:
Hi Greg,
Thanx for the info.
I can count what needs to be written from the elements in iov[]. So that
I can fix
in my code. But if -ERRNO is returned could there be a partial write? Hoi,
When writting a Rados/Rbd client I noticed that the sequence:
r = rbd_aio_readv(ri->ri_image, iov, iovcnt, offset, comp);
r = rbd_aio_wait_for_complete(comp);
nbytes = rbd_aio_get_return_value(comp);
returns the number of bytes read in nbytes.
Now if I do the same but with `rbd_aio_write(v)` I only receive
zero as value...
I would have expected that rbd_aio_get_return_value would more
or less function like aio_return(2):
DESCRIPTION
The aio_return() system call returns the final status of the
asynchronous
I/O request associated with the structure pointed to by iocb.
RETURN VALUES
If the asynchronous I/O request has completed, the status is
returned as
described in read(2), write(2), or fsync(2). Otherwise, aio_return()
returns -1 and sets errno to indicate the error condition.
So I was expecting the amount of bytes written?
Understandable, but for
"distributed systems are hard" reasons, writes
return either 0 (success) or -ERRNO. [1] I guess librbd could
translate that back into the given write size, but it's never come up
as an issue and might have some weird edge cases?
-Greg
[1]: Returning anything else requires recording the return value as an
additional write, or part of the write, so that we give out the same
answer on replay of the write request. We've been talking for years
about extending the protocol+implementation to allow sending back 32
or 64 bytes of data from a write, and...I can't recall if one of those
times it finally got done.
I tried to find something documentation on the client stuff in
librados/rbd, but I
could not find any other than testcode in unittest_librbd. That gives
an idea,
but intricate details are not in there.
--WjW
11 Mar
11 Mar
4:31 p.m.
On Sat, Mar 7, 2020 at 9:45 PM Gregory Farnum <gfarnum(a)redhat.com> wrote:
On Sat, Mar 7, 2020 at 12:39 PM Willem Jan Withagen <wjw(a)digiware.nl> wrote:
It can. Any librbd op that straddles objects would turn into
multiple RADOS ops. This can happen with both small (e.g. 4K)
and big (e.g. 4M) writes even with the default striping parameters.
With clones, the write can be preceded by a bunch of reads from a
parent image and a bunch of copyups. With exclusive-lock, by grabbing
the lock (potentially requesting it from a peer). With object-map,
by updating the object map (potentially twice, before and after the
write-like op, such as discard). The list goes on...
So a partial write can happen in the sense that some parts of your
request can get committed (overwriting old data) even if an error
is returned, and not as in "this many bytes from the beginning of the
request were committed". The entire write must be retried on error.
Thanks,
Ilya
On 7-3-2020 19:13, Gregory Farnum wrote:
If RADOS returns an error code, it won't commit anything.
I'm not sure if librbd can issue single writes which turn into
multiple RADOS writes (by involving multiple objects); I suspect in a
sane configuration it can't but maybe you could coerce it with weird
stripe parameters or block sizes...Jason? On Fri, Mar 6, 2020 at 8:04 AM Willem Jan
Withagen <wjw(a)digiware.nl> wrote:
Hi Greg,
Thanx for the info.
I can count what needs to be written from the elements in iov[]. So that
I can fix
in my code. But if -ERRNO is returned could there be a partial write? Hoi,
When writting a Rados/Rbd client I noticed that the sequence:
r = rbd_aio_readv(ri->ri_image, iov, iovcnt, offset, comp);
r = rbd_aio_wait_for_complete(comp);
nbytes = rbd_aio_get_return_value(comp);
returns the number of bytes read in nbytes.
Now if I do the same but with `rbd_aio_write(v)` I only receive
zero as value...
I would have expected that rbd_aio_get_return_value would more
or less function like aio_return(2):
DESCRIPTION
The aio_return() system call returns the final status of the
asynchronous
I/O request associated with the structure pointed to by iocb.
RETURN VALUES
If the asynchronous I/O request has completed, the status is
returned as
described in read(2), write(2), or fsync(2). Otherwise, aio_return()
returns -1 and sets errno to indicate the error condition.
So I was expecting the amount of bytes written?
Understandable, but for
"distributed systems are hard" reasons, writes
return either 0 (success) or -ERRNO. [1] I guess librbd could
translate that back into the given write size, but it's never come up
as an issue and might have some weird edge cases?
-Greg
[1]: Returning anything else requires recording the return value as an
additional write, or part of the write, so that we give out the same
answer on replay of the write request. We've been talking for years
about extending the protocol+implementation to allow sending back 32
or 64 bytes of data from a write, and...I can't recall if one of those
times it finally got done.
1518
days inactive
1523
days old
4 comments
3 participants
participants (3)
-
Gregory Farnum -
Ilya Dryomov -
Willem Jan Withagen