Using sendfile on Ceph FS results in data stuck in client cache
Hi all, Using sendfile function to write data to cephfs, the data doesn't end up being written. From the client that writes the file, it looks correct at first, but from all other ceph clients, the size is 0 bytes. Re-mounting the filesystem, the data is lost. I didn't see any errors, the data just doesn't get written, as if it's just cached in cephfs client. Writing just an extra byte at the end of the file (without sendfile), it seems to trigger the actual write of all the data. Could someone else confirm if they are also seeing such issue? I'm on ceph 13.2.8, using kernel module for mounting on CentOS7. I've used this sendfile-example for the example below: https://github.com/pijewski/sendfile-example/blob/master/sendfile.c Using a small 27 byte source file. # ls -lh examples/ -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src # ./sendfile examples/src examples/dst 27 # ls -lh examples/ ------x--- 1 root c3-staff 27 Mar 24 18:12 dst -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src But, directory is still on 27 bytes: # ls -lhd examples drwxr-sr-x 1 root c3-staff 27 Mar 24 18:15 examples and on all other cephfs clients, the file is empty: # ls -lh examples/ ------x--- 1 root c3-staff 0 Mar 24 18:12 dst -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src Is this a bug in cephfs, or should I not expect sendfile to work (as it is not posix compliant). There are no error reported from what i can see, and it is 100% reproducible Best regards, Mikael
On Wed, 2020-03-25 at 12:14 +0000, Mikael Öhman wrote:
Hi all,
Using sendfile function to write data to cephfs, the data doesn't end up being written. From the client that writes the file, it looks correct at first, but from all other ceph clients, the size is 0 bytes. Re-mounting the filesystem, the data is lost. I didn't see any errors, the data just doesn't get written, as if it's just cached in cephfs client. Writing just an extra byte at the end of the file (without sendfile), it seems to trigger the actual write of all the data.
Could someone else confirm if they are also seeing such issue? I'm on ceph 13.2.8, using kernel module for mounting on CentOS7.
I've used this sendfile-example for the example below: https://github.com/pijewski/sendfile-example/blob/master/sendfile.c
Using a small 27 byte source file. # ls -lh examples/ -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src # ./sendfile examples/src examples/dst 27 # ls -lh examples/ ------x--- 1 root c3-staff 27 Mar 24 18:12 dst -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src
But, directory is still on 27 bytes: # ls -lhd examples drwxr-sr-x 1 root c3-staff 27 Mar 24 18:15 examples
and on all other cephfs clients, the file is empty: # ls -lh examples/ ------x--- 1 root c3-staff 0 Mar 24 18:12 dst -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src
Is this a bug in cephfs, or should I not expect sendfile to work (as it is not posix compliant). There are no error reported from what i can see, and it is 100% reproducible
(sorry for the resend, the original got caught up in moderation as I sent it from wrong address) This sounds like a kernel client bug in an old Centos7 kernel. The program seems to work as expected on current mainline kernels, and on 3.10.0-1062.1.2.el7.x86_64 (the latest one I had on my client). What kernel version are you using on the client? If you're not on the latest version Centos7 kernel version, then it'd be good to try that and see if it's still reproducible. For the record: [jlayton@centos7 ~]$ sudo umount -f /mnt/cephfs ; sudo mount /mnt/cephfs ; ./sendfile ./testfile /mnt/cephfs/testfile 27 ; sudo umount /mnt/cephfs ; sudo mount /mnt/cephfs ; ls -l /mnt/cephfs Sent 0 KiB over sendfile(3EXT) of 0 KiB requested total 1 drwxr-xr-x 1 root root 1 Mar 25 08:53 foo drwxrwxrwx 1 root root 0 Mar 25 08:37 scratch drwxr-xr-x 1 root root 57 Mar 25 08:44 test -rw-r--r-- 1 jlayton jlayton 27 Mar 25 12:35 testfile [jlayton@centos7 ~]$ uname -r 3.10.0-1062.1.2.el7.x86_64 Attached is the cleaned-up version of sendfile.c that I was using. -- Jeff Layton <jlayton@redhat.com>
Hi Jeff! (also, I'm also sorry for a resend, I did exactly the same with my message as well!) Unfortunately, the answer wasn't that simple, as I am on the latest C7 kernel as well uname -r 3.10.0-1062.1.2.el7.x86_64 I did some more testing, and it's a bit difficult to trigger this reliably when using a line like you do. If I paste a sendfile+remount line like that, it does seem to trigger a proper write, but, if I put some delay in, it fails. About 10 seconds seems to be enough for me; root@hermes:~# /root/sendfile sendfile.c /cephyr/dest 27; sleep 10; umount /cephyr; mount /cephyr; ls -l /cephyr/dest Sent 0 KiB over sendfile(3EXT) of 0 KiB requested -rw-r--r-- 1 root root 0 Mar 25 23:01 /cephyr/dest Easier yet; I can reliable see the problem immediately by not re-mounting anything and just looking at the files from another node; root@hermes:~# /root/sendfile sendfile.c /cephyr/dest 27; ls -l /cephyr/dest Sent 0 KiB over sendfile(3EXT) of 0 KiB requested -rw-r--r-- 1 root root 27 Mar 25 23:10 /cephyr/dest and then, go to another node and look a couple seconds later: [c3-micke@hebbe-c1 ~]$ ls -l /cephyr/dest -rw-r--r-- 1 root root 0 Mar 25 23:10 /cephyr/dest Best regards, Mikael
On Wed, 2020-03-25 at 22:10 +0000, Mikael Öhman wrote:
Hi Jeff! (also, I'm also sorry for a resend, I did exactly the same with my message as well!)
Unfortunately, the answer wasn't that simple, as I am on the latest C7 kernel as well uname -r 3.10.0-1062.1.2.el7.x86_64
I did some more testing, and it's a bit difficult to trigger this reliably when using a line like you do. If I paste a sendfile+remount line like that, it does seem to trigger a proper write, but, if I put some delay in, it fails. About 10 seconds seems to be enough for me;
root@hermes:~# /root/sendfile sendfile.c /cephyr/dest 27; sleep 10; umount /cephyr; mount /cephyr; ls -l /cephyr/dest Sent 0 KiB over sendfile(3EXT) of 0 KiB requested -rw-r--r-- 1 root root 0 Mar 25 23:01 /cephyr/dest
Easier yet; I can reliable see the problem immediately by not re-mounting anything and just looking at the files from another node; root@hermes:~# /root/sendfile sendfile.c /cephyr/dest 27; ls -l /cephyr/dest Sent 0 KiB over sendfile(3EXT) of 0 KiB requested -rw-r--r-- 1 root root 27 Mar 25 23:10 /cephyr/dest and then, go to another node and look a couple seconds later: [c3-micke@hebbe-c1 ~]$ ls -l /cephyr/dest -rw-r--r-- 1 root root 0 Mar 25 23:10 /cephyr/dest
Thanks, I was able to reproduce it this morning. Not sure why the timing matters just yet, but I suspect this will be fixed by the attached patch that's already slated for RHEL7.9 If you're able to build a test a kernel with that patch, then please let us know if it fixes this problem for you as well. Thanks, -- Jeff Layton <jlayton@redhat.com>
Hi Jeff, I got around to building 3.10.0-1062.18.1 with the patch you included, and it seems to be fixed. Thank you very much for your help! Best regards, Mikael
participants (2)
-
Jeff Layton
-
Mikael Öhman