On Wed, 2020-03-25 at 12:14 +0000, Mikael Öhman wrote:
Hi all,
Using sendfile function to write data to cephfs, the data doesn't end up being
written.
From the client that writes the file, it looks correct at first, but from all other ceph
clients, the size is 0 bytes. Re-mounting the filesystem, the data is lost.
I didn't see any errors, the data just doesn't get written, as if it's just
cached in cephfs client.
Writing just an extra byte at the end of the file (without sendfile), it seems to trigger
the actual write of all the data.
Could someone else confirm if they are also seeing such issue? I'm on ceph 13.2.8,
using kernel module for mounting on CentOS7.
I've used this sendfile-example for the example below:
https://github.com/pijewski/sendfile-example/blob/master/sendfile.c
Using a small 27 byte source file.
# ls -lh examples/
-rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src
# ./sendfile examples/src examples/dst 27
# ls -lh examples/
------x--- 1 root c3-staff 27 Mar 24 18:12 dst
-rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src
But, directory is still on 27 bytes:
# ls -lhd examples
drwxr-sr-x 1 root c3-staff 27 Mar 24 18:15 examples
and on all other cephfs clients, the file is empty:
# ls -lh examples/
------x--- 1 root c3-staff 0 Mar 24 18:12 dst
-rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src
Is this a bug in cephfs, or should I not expect sendfile to work (as it is not posix
compliant). There are no error reported from what i can see, and it is 100% reproducible
(sorry for the resend, the original got caught up in moderation as I
sent it from wrong address)
This sounds like a kernel client bug in an old Centos7 kernel. The
program seems to work as expected on current mainline kernels, and on
3.10.0-1062.1.2.el7.x86_64 (the latest one I had on my client).
What kernel version are you using on the client? If you're not on the
latest version Centos7 kernel version, then it'd be good to try that and
see if it's still reproducible. For the record:
[jlayton@centos7 ~]$ sudo umount -f /mnt/cephfs ; sudo mount /mnt/cephfs ; ./sendfile
./testfile /mnt/cephfs/testfile 27 ; sudo umount /mnt/cephfs ; sudo mount /mnt/cephfs ; ls
-l /mnt/cephfs
Sent 0 KiB over sendfile(3EXT) of 0 KiB requested
total 1
drwxr-xr-x 1 root root 1 Mar 25 08:53 foo
drwxrwxrwx 1 root root 0 Mar 25 08:37 scratch
drwxr-xr-x 1 root root 57 Mar 25 08:44 test
-rw-r--r-- 1 jlayton jlayton 27 Mar 25 12:35 testfile
[jlayton@centos7 ~]$ uname -r
3.10.0-1062.1.2.el7.x86_64
Attached is the cleaned-up version of sendfile.c that I was using.
--
Jeff Layton <jlayton(a)redhat.com>