22 Mar
2023
22 Mar
'23
6:14 p.m.
Hi all,
on an NFS re-export of a ceph-fs (kernel client) I observe a very strange error. I'm
un-taring a larger package (1.2G) and after some time I get these errors:
ln: failed to create hard link 'file name': Read-only file system
The strange thing is that this seems only temporary. When I used "ln src dst"
for manual testing, the command failed as above. However, after that I tried "ln -v
src dst" and this command created the hard link with exactly the same path arguments.
During the period when the error occurs, I can't see any FS in read-only mode, neither
on the NFS client nor the NFS server. Funny thing is that file creation and write still
works, its only the hard-link creation that fails.
For details, the set-up is:
file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to other server
other server: mount /shares/path as NFS
More precisely, on the file-server:
fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph
defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev 0 0
exports: /shares/nfs/folder -no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP
On the host at DEST-IP:
fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev 0 0
Both, the file server and the client server are virtual machines. The file server is on
Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine is on AlmaLinux 8
(4.18.0-425.13.1.el8_7.x86_64).
When I change the NFS export from "async" to "sync" everything works.
However, that's a rather bad workaround and not a solution. Although this looks like
an NFS issue, I'm afraid it is a problem with hard links and ceph-fs. It looks like a
race with scheduling and executing operations on the ceph-fs kernel mount.
Has anyone seen something like that?
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
22 Mar
22 Mar
6:42 p.m.
New subject: Advices for the best way to move db/wal lv from old nvme to new one
Hello,
We have a cluster with 26 nodes, and 15 nodes have a bad batch of 2 nvme wheree we have
for each 6 lv for db/wal. We have to change it, because they fail one by one...
The defective nvme are M2 samsung enterprise.
When they fail, we got sense errors, and the nvme disappear, if we power of the server,
and power on, it come back... If we just do a soft reboot, the nvme don’t come back...
So we have decided to replace all of them by Intel pcie SSDPEDME016T4S
The original one are 1Tb, and the new are 1,6Tb
What is the best method to do that ?
put the node in maint mode, and do a pvmove of each and doing after a lv resize of each ?
Or there are a easier way to do that ? Like what I found in mailing list archives ?
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-${OSD}
bluefs-bdev-new-db --dev-target /dev/bluesfs_db/db-osd${OSD}
and
ceph-bluestore-tool --path dev/osd1/ --devs-source dev/osd1/block
--dev-target dev/osd1/block.db bluefs-bdev-migrate
For your understanding, we are in last Quincy and hardware is :
12 x 18tb sas
2 x nvme for db/wal
Thanks for advance for your insights
7:34 p.m.
I have to correct myself. It also fails on an export with "sync" mode. Here is
an strace on the client (strace ln envs/satwindspy/include/ffi.h
mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h):
[...]
stat("mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0x7ffdc5c32820) = -1
ENOENT (No such file or directory)
lstat("envs/satwindspy/include/ffi.h", {st_mode=S_IFREG|0664, st_size=13934,
...}) = 0
linkat(AT_FDCWD, "envs/satwindspy/include/ffi.h", AT_FDCWD,
"mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0) = -1 EROFS (Read-only
file system)
[...]
write(2, "ln: ", 4ln: ) = 4
write(2, "failed to create hard link 'mamb"..., 80failed to create hard link
'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h') = 80
[...]
write(2, ": Read-only file system", 23: Read-only file system) = 23
write(2, "\n", 1
) = 1
lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
close(0) = 0
close(1) = 0
close(2) = 0
exit_group(1) = ?
+++ exited with 1 +++
Has anyone advice?
Thanks!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: Wednesday, March 22, 2023 2:44 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] ln: failed to create hard link 'file name': Read-only file
system
Hi all,
on an NFS re-export of a ceph-fs (kernel client) I observe a very strange error. I'm
un-taring a larger package (1.2G) and after some time I get these errors:
ln: failed to create hard link 'file name': Read-only file system
The strange thing is that this seems only temporary. When I used "ln src dst"
for manual testing, the command failed as above. However, after that I tried "ln -v
src dst" and this command created the hard link with exactly the same path arguments.
During the period when the error occurs, I can't see any FS in read-only mode, neither
on the NFS client nor the NFS server. Funny thing is that file creation and write still
works, its only the hard-link creation that fails.
For details, the set-up is:
file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to other server
other server: mount /shares/path as NFS
More precisely, on the file-server:
fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph
defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev 0 0
exports: /shares/nfs/folder -no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP
On the host at DEST-IP:
fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev 0 0
Both, the file server and the client server are virtual machines. The file server is on
Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine is on AlmaLinux 8
(4.18.0-425.13.1.el8_7.x86_64).
When I change the NFS export from "async" to "sync" everything works.
However, that's a rather bad workaround and not a solution. Although this looks like
an NFS issue, I'm afraid it is a problem with hard links and ceph-fs. It looks like a
race with scheduling and executing operations on the ceph-fs kernel mount.
Has anyone seen something like that?
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:44 p.m.
Do you have logs of what the nfs server is doing?
Managed to reproduce it in terms of direct CephFS ops?
On Wed, Mar 22, 2023 at 8:05 AM Frank Schilder <frans(a)dtu.dk> wrote:
I have to correct myself. It also fails on an export
with "sync" mode.
Here is an strace on the client (strace ln envs/satwindspy/include/ffi.h
mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h):
[...]
stat("mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h",
0x7ffdc5c32820) = -1 ENOENT (No such file or directory)
lstat("envs/satwindspy/include/ffi.h", {st_mode=S_IFREG|0664,
st_size=13934, ...}) = 0
linkat(AT_FDCWD, "envs/satwindspy/include/ffi.h", AT_FDCWD,
"mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0) = -1 EROFS
(Read-only file system)
[...]
write(2, "ln: ", 4ln: ) = 4
write(2, "failed to create hard link 'mamb"..., 80failed to create hard
link 'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h') = 80
[...]
write(2, ": Read-only file system", 23: Read-only file system) = 23
write(2, "\n", 1
) = 1
lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
close(0) = 0
close(1) = 0
close(2) = 0
exit_group(1) = ?
+++ exited with 1 +++
Has anyone advice?
Thanks!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: Wednesday, March 22, 2023 2:44 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] ln: failed to create hard link 'file name':
Read-only file system
Hi all,
on an NFS re-export of a ceph-fs (kernel client) I observe a very strange
error. I'm un-taring a larger package (1.2G) and after some time I get
these errors:
ln: failed to create hard link 'file name': Read-only file system
The strange thing is that this seems only temporary. When I used "ln src
dst" for manual testing, the command failed as above. However, after that I
tried "ln -v src dst" and this command created the hard link with exactly
the same path arguments. During the period when the error occurs, I can't
see any FS in read-only mode, neither on the NFS client nor the NFS server.
Funny thing is that file creation and write still works, its only the
hard-link creation that fails.
For details, the set-up is:
file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to
other server
other server: mount /shares/path as NFS
More precisely, on the file-server:
fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph
defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev
0 0
exports: /shares/nfs/folder
-no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP
On the host at DEST-IP:
fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev
0 0
Both, the file server and the client server are virtual machines. The file
server is on Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine
is on AlmaLinux 8 (4.18.0-425.13.1.el8_7.x86_64).
When I change the NFS export from "async" to "sync" everything
works.
However, that's a rather bad workaround and not a solution. Although this
looks like an NFS issue, I'm afraid it is a problem with hard links and
ceph-fs. It looks like a race with scheduling and executing operations on
the ceph-fs kernel mount.
Has anyone seen something like that?
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:57 p.m.
Hi Gregory,
thanks for your reply. First a quick update. Here is how I get ln to work after it failed,
there seems no timeout:
$ ln envs/satwindspy/include/ffi.h mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
ln: failed to create hard link
'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h': Read-only file system
$ ls -l envs/satwindspy/include mambaforge/pkgs/libffi-3.3-h58526e2_2
envs/satwindspy/include:
total 7664
-rw-rw-r--. 1 rit rit 959 Mar 5 2021 ares_build.h
[...]
$ ln envs/satwindspy/include/ffi.h mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
After an ls -l on both directories ln works.
To the question: How can I pull out a log from the nfs server? There is nothing in
/var/log/messages.
I can't reproduce it with simple commands on the NFS client. It seems to occur only
when a large number of files/dirs is created. I can make the archive available to you if
this helps.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Gregory Farnum <gfarnum(a)redhat.com>
Sent: Wednesday, March 22, 2023 4:14 PM
To: Frank Schilder
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
Do you have logs of what the nfs server is doing?
Managed to reproduce it in terms of direct CephFS ops?
On Wed, Mar 22, 2023 at 8:05 AM Frank Schilder
<frans@dtu.dk<mailto:frans@dtu.dk>> wrote:
I have to correct myself. It also fails on an export with "sync" mode. Here is
an strace on the client (strace ln envs/satwindspy/include/ffi.h
mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h):
[...]
stat("mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0x7ffdc5c32820) = -1
ENOENT (No such file or directory)
lstat("envs/satwindspy/include/ffi.h", {st_mode=S_IFREG|0664, st_size=13934,
...}) = 0
linkat(AT_FDCWD, "envs/satwindspy/include/ffi.h", AT_FDCWD,
"mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0) = -1 EROFS (Read-only
file system)
[...]
write(2, "ln: ", 4ln: ) = 4
write(2, "failed to create hard link 'mamb"..., 80failed to create hard link
'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h') = 80
[...]
write(2, ": Read-only file system", 23: Read-only file system) = 23
write(2, "\n", 1
) = 1
lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
close(0) = 0
close(1) = 0
close(2) = 0
exit_group(1) = ?
+++ exited with 1 +++
Has anyone advice?
Thanks!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>>
Sent: Wednesday, March 22, 2023 2:44 PM
To: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: [ceph-users] ln: failed to create hard link 'file name': Read-only file
system
Hi all,
on an NFS re-export of a ceph-fs (kernel client) I observe a very strange error. I'm
un-taring a larger package (1.2G) and after some time I get these errors:
ln: failed to create hard link 'file name': Read-only file system
The strange thing is that this seems only temporary. When I used "ln src dst"
for manual testing, the command failed as above. However, after that I tried "ln -v
src dst" and this command created the hard link with exactly the same path arguments.
During the period when the error occurs, I can't see any FS in read-only mode, neither
on the NFS client nor the NFS server. Funny thing is that file creation and write still
works, its only the hard-link creation that fails.
For details, the set-up is:
file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to other server
other server: mount /shares/path as NFS
More precisely, on the file-server:
fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph
defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev 0 0
exports: /shares/nfs/folder -no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP
On the host at DEST-IP:
fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev 0 0
Both, the file server and the client server are virtual machines. The file server is on
Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine is on AlmaLinux 8
(4.18.0-425.13.1.el8_7.x86_64).
When I change the NFS export from "async" to "sync" everything works.
However, that's a rather bad workaround and not a solution. Although this looks like
an NFS issue, I'm afraid it is a problem with hard links and ceph-fs. It looks like a
race with scheduling and executing operations on the ceph-fs kernel mount.
Has anyone seen something like that?
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
8:11 p.m.
On Wed, Mar 22, 2023 at 8:27 AM Frank Schilder <frans(a)dtu.dk> wrote:
Hi Gregory,
thanks for your reply. First a quick update. Here is how I get ln to work
after it failed, there seems no timeout:
$ ln envs/satwindspy/include/ffi.h
mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
ln: failed to create hard link
'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h': Read-only file system
$ ls -l envs/satwindspy/include mambaforge/pkgs/libffi-3.3-h58526e2_2
envs/satwindspy/include:
total 7664
-rw-rw-r--. 1 rit rit 959 Mar 5 2021 ares_build.h
[...]
$ ln envs/satwindspy/include/ffi.h
mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
After an ls -l on both directories ln works.
To the question: How can I pull out a log from the nfs server? There is
nothing in /var/log/messages.
So you’re using the kernel server and re-exporting, right?
I’m not very familiar with its implementation; I wonder if it’s doing
something strange via the kernel vfs.
AFAIK this isn’t really supportable for general use because nfs won’t
respect the CephFS file consistency protocol. But maybe it’s trying a bit
and that’s causing trouble?
-Greg
I can't reproduce it with simple commands on the NFS client. It seems to
occur only when a large number of files/dirs is created. I can make the
archive available to you if this helps.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Gregory Farnum <gfarnum(a)redhat.com>
Sent: Wednesday, March 22, 2023 4:14 PM
To: Frank Schilder
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
Do you have logs of what the nfs server is doing?
Managed to reproduce it in terms of direct CephFS ops?
On Wed, Mar 22, 2023 at 8:05 AM Frank Schilder <frans(a)dtu.dk<mailtomailto:
frans(a)dtu.dk>> wrote:
I have to correct myself. It also fails on an export with "sync" mode.
Here is an strace on the client (strace ln envs/satwindspy/include/ffi.h
mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h):
[...]
stat("mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h",
0x7ffdc5c32820) = -1 ENOENT (No such file or directory)
lstat("envs/satwindspy/include/ffi.h", {st_mode=S_IFREG|0664,
st_size=13934, ...}) = 0
linkat(AT_FDCWD, "envs/satwindspy/include/ffi.h", AT_FDCWD,
"mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0) = -1 EROFS
(Read-only file system)
[...]
write(2, "ln: ", 4ln: ) = 4
write(2, "failed to create hard link 'mamb"..., 80failed to create hard
link 'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h') = 80
[...]
write(2, ": Read-only file system", 23: Read-only file system) = 23
write(2, "\n", 1
) = 1
lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
close(0) = 0
close(1) = 0
close(2) = 0
exit_group(1) = ?
+++ exited with 1 +++
Has anyone advice?
Thanks!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>>
Sent: Wednesday, March 22, 2023 2:44 PM
To: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: [ceph-users] ln: failed to create hard link 'file name':
Read-only file system
Hi all,
on an NFS re-export of a ceph-fs (kernel client) I observe a very strange
error. I'm un-taring a larger package (1.2G) and after some time I get
these errors:
ln: failed to create hard link 'file name': Read-only file system
The strange thing is that this seems only temporary. When I used "ln src
dst" for manual testing, the command failed as above. However, after that I
tried "ln -v src dst" and this command created the hard link with exactly
the same path arguments. During the period when the error occurs, I can't
see any FS in read-only mode, neither on the NFS client nor the NFS server.
Funny thing is that file creation and write still works, its only the
hard-link creation that fails.
For details, the set-up is:
file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to
other server
other server: mount /shares/path as NFS
More precisely, on the file-server:
fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph
defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev
0 0
exports: /shares/nfs/folder
-no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP
On the host at DEST-IP:
fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev
0 0
Both, the file server and the client server are virtual machines. The file
server is on Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine
is on AlmaLinux 8 (4.18.0-425.13.1.el8_7.x86_64).
When I change the NFS export from "async" to "sync" everything
works.
However, that's a rather bad workaround and not a solution. Although this
looks like an NFS issue, I'm afraid it is a problem with hard links and
ceph-fs. It looks like a race with scheduling and executing operations on
the ceph-fs kernel mount.
Has anyone seen something like that?
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailtolto:
ceph-users-leave(a)ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailtolto:
ceph-users-leave(a)ceph.io>
28 Mar
28 Mar
11:18 a.m.
On 22/03/2023 23:41, Gregory Farnum wrote:
On Wed, Mar 22, 2023 at 8:27 AM Frank Schilder
<frans(a)dtu.dk> wrote:
Yeah, I think you are right Greg.
Checked the logs uploaded by Frank. I found that the kclient just send
one request like:
++++++
2023-03-27T23:24:37.866+0200 7f0c1a0d1700 7 mds.0.server
dispatch_client_request client_request(client.186555:475421 link
#0x10000682337/liblz4.so.1.9.3 #0x1000066d6d8//
2023-03-27T23:24:37.864907+0200 caller_uid=1000,
caller_gid=1000{4,24,27,30,46,122,134,135,1000,}) v4
2023-03-27T23:24:37.866+0200 7f0c1a0d1700 7 mds.0.server
handle_client_link #0x10000682337/liblz4.so.1.9.3 to #0x1000066d6d8//
2023-03-27T23:24:37.866+0200 7f0c1a0d1700 10 mds.0.server
rdlock_two_paths_xlock_destdn request(client.186555:475421 nref=2
cr=0x5601bbc60500) #0x10000682337/liblz4.so.1.9.3 #0x1000066d6d8//
2023-03-27T23:24:37.866+0200 7f0c1a0d1700 7 mds.0.server
reply_client_request -30 ((30) Read-only file system)
client_request(client.186555:475421 link #0x10000682337/liblz4.so.1.9.3
#0x1000066d6d8// 2023-03-27T23:24:37.864907+0200 caller_uid=1000,
caller_gid=1000{4,24,27,30,46,122,134,135,1000,}) v4
------
The kclient just set the src dentry to "#0x1000066d6d8//". While the mds
will parse the "//" as a snapdir, which is readonly. This is why mds
return a -EROFS error.
But from mds logs we can see that the "0x1000066d6d8" is
"/data/nfs/envs/satwindspy/lib/liblz4.so.1.9.3":
++++++
2023-03-27T23:24:37.866+0200 7f0c1a0d1700 7 mds.0.locker issue_caps
allowed=pAsLsXsFscrl, xlocker allowed=pAsLsXsFscrl on [inode
0x1000066d6d8 [...7b,head] /data/nfs/envs/satwindspy/lib/liblz4.so.1.9.3
auth v7035 snaprealm=0x55fe3785e500 s=215880 nl=2 n(v0
rc2023-03-27T23:15:22.568391+0200 b215880 1=1+0) (iversion lock)
caps={186555=pAsXsFscr/-@3} | ptrwaiter=0 request=0 lock=0 caps=1
remoteparent=1 dirtyparent=0 dirty=0 authpin=0 0x5601b7174800]
------
Then from the kernel debug logs:
++++++
31358125 [16380611.812642] ceph: do_request mds0 session
00000000a66983cb state open
31358126 [16380611.812644] ceph: __prepare_send_request
000000001ebc34fd tid 475421 link (attempt 1)
31358127 [16380611.812647] ceph: dentry 000000006cbb0f2e
10000682337/liblz4.so.1.9.3
31358128 [16380611.812649] ceph: dentry 00000000126d4660 1000066d6d8//
------
We can see that the kclient set the src dentry to "1000066d6d8//".
This is incorrect and it should be "1000066d2e3/liblz4.so.1.9.3", which
the "1000066d2e3" is the parent dir's inode and the path is
"/data/nfs/envs/satwindspy/lib/".
From the fs/ceph/dir.c code, we can see that the ceph_link() will parse
the src dentry:
2735 static int build_dentry_path(struct dentry *dentry, struct inode *dir,
2736 const char **ppath, int *ppathlen, u64
*pino,
2737 bool *pfreepath, bool parent_locked)
2738 {
2739 char *path;
2740
2741 rcu_read_lock();
2742 if (!dir)
2743 dir = d_inode_rcu(dentry->d_parent);
2744 if (dir && parent_locked && ceph_snap(dir) == CEPH_NOSNAP
&& !IS_ENCRYPTED(dir)) {
2745 *pino = ceph_ino(dir);
2746 rcu_read_unlock();
2747 *ppath = dentry->d_name.name;
2748 *ppathlen = dentry->d_name.len;
2749 return 0;
2750 }
2751 rcu_read_unlock();
2752 path = ceph_mdsc_build_path(dentry, ppathlen, pino, 1);
2753 if (IS_ERR(path))
2754 return PTR_ERR(path);
2755 *ppath = path;
2756 *pfreepath = true;
2757 return 0;
2758 }
In Line#2743, the 'dir' was parsed as "liblz4.so.1.9.3" 's ino#
"1000066d6d8", which is incorrect and it should be the parent dir's ino#
"1000066d2e3". And in Line#2747 the "ppath" is "/", which is
also
incorrect and it should be "liblz4.so.1.9.3".
That means the nfs client passed a invalidate or corrupted old_dentry to
kernel ceph. I have no idea how that could happen.
@Frank,
Could you check the nfs client logs ?
Thanks,
- Xiubo
Hi Gregory,
thanks for your reply. First a quick update. Here is how I get ln to work
after it failed, there seems no timeout:
$ ln envs/satwindspy/include/ffi.h
mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
ln: failed to create hard link
'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h': Read-only file system
$ ls -l envs/satwindspy/include mambaforge/pkgs/libffi-3.3-h58526e2_2
envs/satwindspy/include:
total 7664
-rw-rw-r--. 1 rit rit 959 Mar 5 2021 ares_build.h
[...]
$ ln envs/satwindspy/include/ffi.h
mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
After an ls -l on both directories ln works.
To the question: How can I pull out a log from the nfs server? There is
nothing in /var/log/messages.
So you’re using the kernel server and re-exporting, right?
I’m not very familiar with its implementation; I wonder if it’s doing
something strange via the kernel vfs.
AFAIK this isn’t really supportable for general use because nfs won’t
respect the CephFS file consistency protocol. But maybe it’s trying a bit
and that’s causing trouble? -Greg
--
Best Regards,
Xiubo Li (李秀波)
Email: xiubli@redhat.com/xiubli@ibm.com
Slack: @Xiubo Li
I can't reproduce it with simple commands on
the NFS client. It seems to
occur only when a large number of files/dirs is created. I can make the
archive available to you if this helps.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Gregory Farnum <gfarnum(a)redhat.com>
Sent: Wednesday, March 22, 2023 4:14 PM
To: Frank Schilder
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
Do you have logs of what the nfs server is doing?
Managed to reproduce it in terms of direct CephFS ops?
On Wed, Mar 22, 2023 at 8:05 AM Frank Schilder <frans(a)dtu.dk<mailtomailto:
frans(a)dtu.dk>> wrote:
I have to correct myself. It also fails on an export with "sync" mode.
Here is an strace on the client (strace ln envs/satwindspy/include/ffi.h
mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h):
[...]
stat("mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h",
0x7ffdc5c32820) = -1 ENOENT (No such file or directory)
lstat("envs/satwindspy/include/ffi.h", {st_mode=S_IFREG|0664,
st_size=13934, ...}) = 0
linkat(AT_FDCWD, "envs/satwindspy/include/ffi.h", AT_FDCWD,
"mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0) = -1 EROFS
(Read-only file system)
[...]
write(2, "ln: ", 4ln: ) = 4
write(2, "failed to create hard link 'mamb"..., 80failed to create hard
link 'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h') = 80
[...]
write(2, ": Read-only file system", 23: Read-only file system) = 23
write(2, "\n", 1
) = 1
lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
close(0) = 0
close(1) = 0
close(2) = 0
exit_group(1) = ?
+++ exited with 1 +++
Has anyone advice?
Thanks!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>>
Sent: Wednesday, March 22, 2023 2:44 PM
To: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: [ceph-users] ln: failed to create hard link 'file name':
Read-only file system
Hi all,
on an NFS re-export of a ceph-fs (kernel client) I observe a very strange
error. I'm un-taring a larger package (1.2G) and after some time I get
these errors:
ln: failed to create hard link 'file name': Read-only file system
The strange thing is that this seems only temporary. When I used "ln src
dst" for manual testing, the command failed as above. However, after that I
tried "ln -v src dst" and this command created the hard link with exactly
the same path arguments. During the period when the error occurs, I can't
see any FS in read-only mode, neither on the NFS client nor the NFS server.
Funny thing is that file creation and write still works, its only the
hard-link creation that fails.
For details, the set-up is:
file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to
other server
other server: mount /shares/path as NFS
More precisely, on the file-server:
fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph
defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev
0 0
exports: /shares/nfs/folder
-no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP
On the host at DEST-IP:
fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev
0 0
Both, the file server and the client server are virtual machines. The file
server is on Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine
is on AlmaLinux 8 (4.18.0-425.13.1.el8_7.x86_64).
When I change the NFS export from "async" to "sync" everything
works.
However, that's a rather bad workaround and not a solution. Although this
looks like an NFS issue, I'm afraid it is a problem with hard links and
ceph-fs. It looks like a race with scheduling and executing operations on
the ceph-fs kernel mount.
Has anyone seen something like that?
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailtolto:
ceph-users-leave(a)ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailtolto:
ceph-users-leave(a)ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
23 Mar
23 Mar
6:11 a.m.
Hi Frank,
Could you reproduce it again by enabling the kclient debug logs and also
the mds debug logs ?
I need to know what exactly has happened in kclient and mds side.
Locally I couldn't reproduce it.
Thanks
- Xiubo
On 22/03/2023 23:27, Frank Schilder wrote:
Hi Gregory,
thanks for your reply. First a quick update. Here is how I get ln to work after it
failed, there seems no timeout:
$ ln envs/satwindspy/include/ffi.h mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
ln: failed to create hard link
'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h': Read-only file system
$ ls -l envs/satwindspy/include mambaforge/pkgs/libffi-3.3-h58526e2_2
envs/satwindspy/include:
total 7664
-rw-rw-r--. 1 rit rit 959 Mar 5 2021 ares_build.h
[...]
$ ln envs/satwindspy/include/ffi.h mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
After an ls -l on both directories ln works.
To the question: How can I pull out a log from the nfs server? There is nothing in
/var/log/messages.
I can't reproduce it with simple commands on the NFS client. It seems to occur only
when a large number of files/dirs is created. I can make the archive available to you if
this helps.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Gregory Farnum <gfarnum(a)redhat.com>
Sent: Wednesday, March 22, 2023 4:14 PM
To: Frank Schilder
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
Do you have logs of what the nfs server is doing?
Managed to reproduce it in terms of direct CephFS ops?
On Wed, Mar 22, 2023 at 8:05 AM Frank Schilder
<frans@dtu.dk<mailto:frans@dtu.dk>> wrote:
I have to correct myself. It also fails on an export with "sync" mode. Here is
an strace on the client (strace ln envs/satwindspy/include/ffi.h
mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h):
[...]
stat("mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0x7ffdc5c32820) =
-1 ENOENT (No such file or directory)
lstat("envs/satwindspy/include/ffi.h", {st_mode=S_IFREG|0664, st_size=13934,
...}) = 0
linkat(AT_FDCWD, "envs/satwindspy/include/ffi.h", AT_FDCWD,
"mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0) = -1 EROFS (Read-only
file system)
[...]
write(2, "ln: ", 4ln: ) = 4
write(2, "failed to create hard link 'mamb"..., 80failed to create hard
link 'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h') = 80
[...]
write(2, ": Read-only file system", 23: Read-only file system) = 23
write(2, "\n", 1
) = 1
lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
close(0) = 0
close(1) = 0
close(2) = 0
exit_group(1) = ?
+++ exited with 1 +++
Has anyone advice?
Thanks!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>>
Sent: Wednesday, March 22, 2023 2:44 PM
To: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: [ceph-users] ln: failed to create hard link 'file name': Read-only file
system
Hi all,
on an NFS re-export of a ceph-fs (kernel client) I observe a very strange error. I'm
un-taring a larger package (1.2G) and after some time I get these errors:
ln: failed to create hard link 'file name': Read-only file system
The strange thing is that this seems only temporary. When I used "ln src dst"
for manual testing, the command failed as above. However, after that I tried "ln -v
src dst" and this command created the hard link with exactly the same path arguments.
During the period when the error occurs, I can't see any FS in read-only mode, neither
on the NFS client nor the NFS server. Funny thing is that file creation and write still
works, its only the hard-link creation that fails.
For details, the set-up is:
file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to other server
other server: mount /shares/path as NFS
More precisely, on the file-server:
fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph
defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev 0 0
exports: /shares/nfs/folder -no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP
On the host at DEST-IP:
fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev 0 0
Both, the file server and the client server are virtual machines. The file server is on
Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine is on AlmaLinux 8
(4.18.0-425.13.1.el8_7.x86_64).
When I change the NFS export from "async" to "sync" everything works.
However, that's a rather bad workaround and not a solution. Although this looks like
an NFS issue, I'm afraid it is a problem with hard links and ceph-fs. It looks like a
race with scheduling and executing operations on the ceph-fs kernel mount.
Has anyone seen something like that?
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Best Regards,
Xiubo Li (李秀波)
Email: xiubli@redhat.com/xiubli@ibm.com
Slack: @Xiubo Li
10:26 p.m.
Hi Xiubo and Gregory,
sorry for the slow reply, I did some more debugging and didn't have too much time.
First some questions to collecting logs, but please see also below for reproducing the
issue yourselves.
I can reproduce it reliably but need some input for these:
enabling the kclient debug logs and
How do I do that? I thought the kclient ignores the ceph.conf and I'm not aware
of a mount option to this effect. Is there a "ceph config set ..." setting I can
change for a specific client (by host name/IP) and how exactly?
also the mds debug logs
I guess here I should set a higher loglevel for the MDS serving this directory (it
is pinned to a single rank) or is it something else?
The issue seems to require a certain load to show up. I created a minimal tar file
mimicking the problem and having 2 directories with a hard link from a file in the first
to a new name in the second directory. This does not cause any problems, so its not that
easy to reproduce.
How you can reproduce it:
As an alternative to my limited skills of pulling logs out, I make the tgz-archive
available to you both. You will receive an e-mail from our one-drive with a download link.
If you un-tar the archive on an NFS client dir that's a re-export of a kclient mount,
after some time you should see the errors showing up.
I can reliably reproduce these errors on our production- as well as on our test cluster.
You should be able to reproduce it too with the tgz file.
Here is a result on our set-up:
- production cluster (executed in a sub-dir conda to make cleanup easy):
$ time tar -xzf ../conda.tgz
tar: mambaforge/pkgs/libstdcxx-ng-9.3.0-h6de172a_18/lib/libstdc++.so.6.0.28: Cannot hard
link to ‘envs/satwindspy/lib/libstdc++.so.6.0.28’: Read-only file system
[...]
tar: mambaforge/pkgs/boost-cpp-1.72.0-h9d3c048_4/lib/libboost_log.so.1.72.0: Cannot hard
link to ‘envs/satwindspy/lib/libboost_log.so.1.72.0’: Read-only file system
^C
real 1m29.008s
user 0m0.612s
sys 0m6.870s
By this time there are already hard links created, so it doesn't fail right away:
$ find -type f -links +1
./mambaforge/pkgs/libev-4.33-h516909a_1/share/man/man3/ev.3
./mambaforge/pkgs/libev-4.33-h516909a_1/include/ev++.h
./mambaforge/pkgs/libev-4.33-h516909a_1/include/ev.h
...
- test cluster (octopus latest stable, 3 OSD hosts with 3 HDD OSDs each, simple
ceph-fs):
# ceph fs status
fs - 2 clients
==
RANK STATE MDS ACTIVITY DNS INOS
0 active tceph-02 Reqs: 0 /s 1807k 1739k
POOL TYPE USED AVAIL
fs-meta1 metadata 18.3G 156G
fs-meta2 data 0 156G
fs-data data 1604G 312G
STANDBY MDS
tceph-01
tceph-03
MDS version: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus
(stable)
Its the new recommended 3-pool layout with fs-data being a 4+2 EC pool.
$ time tar -xzf / ... /conda.tgz
tar: mambaforge/ssl/cacert.pem: Cannot hard link to ‘envs/satwindspy/ssl/cacert.pem’:
Read-only file system
[...]
tar: mambaforge/lib/engines-1.1/padlock.so: Cannot hard link to
‘envs/satwindspy/lib/engines-1.1/padlock.so’: Read-only file system
^C
real 6m23.522s
user 0m3.477s
sys 0m25.792s
Same story here, a large number of hard links has already been created before it starts
failing:
$ find -type f -links +1
./mambaforge/lib/liblzo2.so.2.0.0
...
Looking at the output of find in both cases it also looks a bit non-deterministic when it
starts failing.
It would be great if you can reproduce the issue on a similar test setup using the archive
conda.tgz. If not, I'm happy to collect any type of logs on our test cluster.
We have now one user who has problems with rsync to an NFS share and it would be really
appreciated if this could be sorted.
Thanks for your help and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Xiubo Li <xiubli(a)redhat.com>
Sent: Thursday, March 23, 2023 2:41 AM
To: Frank Schilder; Gregory Farnum
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
Hi Frank,
Could you reproduce it again by enabling the kclient debug logs and also
the mds debug logs ?
I need to know what exactly has happened in kclient and mds side.
Locally I couldn't reproduce it.
Thanks
- Xiubo
On 22/03/2023 23:27, Frank Schilder wrote:
Hi Gregory,
thanks for your reply. First a quick update. Here is how I get ln to work after it
failed, there seems no timeout:
$ ln envs/satwindspy/include/ffi.h mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
ln: failed to create hard link
'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h': Read-only file system
$ ls -l envs/satwindspy/include mambaforge/pkgs/libffi-3.3-h58526e2_2
envs/satwindspy/include:
total 7664
-rw-rw-r--. 1 rit rit 959 Mar 5 2021 ares_build.h
[...]
$ ln envs/satwindspy/include/ffi.h mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
After an ls -l on both directories ln works.
To the question: How can I pull out a log from the nfs server? There is nothing in
/var/log/messages.
I can't reproduce it with simple commands on the NFS client. It seems to occur only
when a large number of files/dirs is created. I can make the archive available to you if
this helps.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Gregory Farnum <gfarnum(a)redhat.com>
Sent: Wednesday, March 22, 2023 4:14 PM
To: Frank Schilder
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
Do you have logs of what the nfs server is doing?
Managed to reproduce it in terms of direct CephFS ops?
On Wed, Mar 22, 2023 at 8:05 AM Frank Schilder
<frans@dtu.dk<mailto:frans@dtu.dk>> wrote:
I have to correct myself. It also fails on an export with "sync" mode. Here is
an strace on the client (strace ln envs/satwindspy/include/ffi.h
mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h):
[...]
stat("mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0x7ffdc5c32820) =
-1 ENOENT (No such file or directory)
lstat("envs/satwindspy/include/ffi.h", {st_mode=S_IFREG|0664, st_size=13934,
...}) = 0
linkat(AT_FDCWD, "envs/satwindspy/include/ffi.h", AT_FDCWD,
"mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0) = -1 EROFS (Read-only
file system)
[...]
write(2, "ln: ", 4ln: ) = 4
write(2, "failed to create hard link 'mamb"..., 80failed to create hard
link 'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h') = 80
[...]
write(2, ": Read-only file system", 23: Read-only file system) = 23
write(2, "\n", 1
) = 1
lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
close(0) = 0
close(1) = 0
close(2) = 0
exit_group(1) = ?
+++ exited with 1 +++
Has anyone advice?
Thanks!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>>
Sent: Wednesday, March 22, 2023 2:44 PM
To: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: [ceph-users] ln: failed to create hard link 'file name': Read-only file
system
Hi all,
on an NFS re-export of a ceph-fs (kernel client) I observe a very strange error. I'm
un-taring a larger package (1.2G) and after some time I get these errors:
ln: failed to create hard link 'file name': Read-only file system
The strange thing is that this seems only temporary. When I used "ln src dst"
for manual testing, the command failed as above. However, after that I tried "ln -v
src dst" and this command created the hard link with exactly the same path arguments.
During the period when the error occurs, I can't see any FS in read-only mode, neither
on the NFS client nor the NFS server. Funny thing is that file creation and write still
works, its only the hard-link creation that fails.
For details, the set-up is:
file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to other server
other server: mount /shares/path as NFS
More precisely, on the file-server:
fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph
defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev 0 0
exports: /shares/nfs/folder -no_root_squash,rw,async,mountpoint,no_subtree_check
DEST-IP
On the host at DEST-IP:
fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev 0 0
Both, the file server and the client server are virtual machines. The file server is on
Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine is on AlmaLinux 8
(4.18.0-425.13.1.el8_7.x86_64).
When I change the NFS export from "async" to "sync" everything works.
However, that's a rather bad workaround and not a solution. Although this looks like
an NFS issue, I'm afraid it is a problem with hard links and ceph-fs. It looks like a
race with scheduling and executing operations on the ceph-fs kernel mount.
Has anyone seen something like that?
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Best Regards,
Xiubo Li (李秀波)
Email: xiubli@redhat.com/xiubli@ibm.com
Slack: @Xiubo Li
27 Mar
27 Mar
6:45 p.m.
Frank,
Sorry for late.
On 24/03/2023 01:56, Frank Schilder wrote:
$ ceph daemon mds.X config set debug_mds 25
$ ceph daemon mds.X config set debug_ms 1
Hi Xiubo and Gregory,
sorry for the slow reply, I did some more debugging and didn't have too much time.
First some questions to collecting logs, but please see also below for reproducing the
issue yourselves.
I can reproduce it reliably but need some input for these:
$ echo "module ceph +p" > /sys/kernel/debug/dynamic_debug/control
This will enable the debug logs in kernel ceph. Then please provide the
message logs.
enabling the kclient debug logs and
How do
I do that? I thought the kclient ignores the ceph.conf and I'm not aware of a mount
option to this effect. Is there a "ceph config set ..." setting I can change for
a specific client (by host name/IP) and how exactly?
also the mds
debug logs
I guess here I should set a higher loglevel for the MDS serving this
directory (it is pinned to a single rank) or is it something else?
The issue seems to require a certain load to show up. I created a minimal tar file
mimicking the problem and having 2 directories with a hard link from a file in the first
to a new name in the second directory. This does not cause any problems, so its not that
easy to reproduce.
How you can reproduce it:
As an alternative to my limited skills of pulling logs out, I make the tgz-archive
available to you both. You will receive an e-mail from our one-drive with a download link.
If you un-tar the archive on an NFS client dir that's a re-export of a kclient mount,
after some time you should see the errors showing up.
I can reliably reproduce these errors on our production- as well as on our test cluster.
You should be able to reproduce it too with the tgz file.
Here is a result on our set-up:
- production cluster (executed in a sub-dir conda to make cleanup easy):
$ time tar -xzf ../conda.tgz
tar: mambaforge/pkgs/libstdcxx-ng-9.3.0-h6de172a_18/lib/libstdc++.so.6.0.28: Cannot hard
link to ‘envs/satwindspy/lib/libstdc++.so.6.0.28’: Read-only file system
[...]
tar: mambaforge/pkgs/boost-cpp-1.72.0-h9d3c048_4/lib/libboost_log.so.1.72.0: Cannot hard
link to ‘envs/satwindspy/lib/libboost_log.so.1.72.0’: Read-only file system
^C
real 1m29.008s
user 0m0.612s
sys 0m6.870s
By this time there are already hard links created, so it doesn't fail right away:
$ find -type f -links +1
./mambaforge/pkgs/libev-4.33-h516909a_1/share/man/man3/ev.3
./mambaforge/pkgs/libev-4.33-h516909a_1/include/ev++.h
./mambaforge/pkgs/libev-4.33-h516909a_1/include/ev.h
...
- test cluster (octopus latest stable, 3 OSD hosts with 3 HDD OSDs each, simple
ceph-fs):
# ceph fs status
fs - 2 clients
==
RANK STATE MDS ACTIVITY DNS INOS
0 active tceph-02 Reqs: 0 /s 1807k 1739k
POOL TYPE USED AVAIL
fs-meta1 metadata 18.3G 156G
fs-meta2 data 0 156G
fs-data data 1604G 312G
STANDBY MDS
tceph-01
tceph-03
MDS version: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus
(stable)
Its the new recommended 3-pool layout with fs-data being a 4+2 EC pool.
$ time tar -xzf / ... /conda.tgz
tar: mambaforge/ssl/cacert.pem: Cannot hard link to ‘envs/satwindspy/ssl/cacert.pem’:
Read-only file system
[...]
tar: mambaforge/lib/engines-1.1/padlock.so: Cannot hard link to
‘envs/satwindspy/lib/engines-1.1/padlock.so’: Read-only file system
^C
real 6m23.522s
user 0m3.477s
sys 0m25.792s
Same story here, a large number of hard links has already been created before it starts
failing:
$ find -type f -links +1
./mambaforge/lib/liblzo2.so.2.0.0
...
Looking at the output of find in both cases it also looks a bit non-deterministic when it
starts failing.
It would be great if you can reproduce the issue on a similar test setup using the
archive conda.tgz. If not, I'm happy to collect any type of logs on our test cluster.
We have now one user who has problems with rsync to an NFS share and it would be really
appreciated if this could be sorted.
The ceph qa teuthology test cases have already one similar test, which
will untar a kernel tarball, but never seen this yet.
I will try this again tomorrow without the NFS client.
Thanks
- Xiubo
Thanks for your help and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Xiubo Li <xiubli(a)redhat.com>
Sent: Thursday, March 23, 2023 2:41 AM
To: Frank Schilder; Gregory Farnum
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
Hi Frank,
Could you reproduce it again by enabling the kclient debug logs and also
the mds debug logs ?
I need to know what exactly has happened in kclient and mds side.
Locally I couldn't reproduce it.
Thanks
- Xiubo
On 22/03/2023 23:27, Frank Schilder wrote:
--
Best Regards,
Xiubo Li (李秀波)
Email: xiubli@redhat.com/xiubli@ibm.com
Slack: @Xiubo Li
Hi Gregory,
thanks for your reply. First a quick update. Here is how I get ln to work after it
failed, there seems no timeout:
$ ln envs/satwindspy/include/ffi.h mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
ln: failed to create hard link
'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h': Read-only file system
$ ls -l envs/satwindspy/include mambaforge/pkgs/libffi-3.3-h58526e2_2
envs/satwindspy/include:
total 7664
-rw-rw-r--. 1 rit rit 959 Mar 5 2021 ares_build.h
[...]
$ ln envs/satwindspy/include/ffi.h mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
After an ls -l on both directories ln works.
To the question: How can I pull out a log from the nfs server? There is nothing in
/var/log/messages.
I can't reproduce it with simple commands on the NFS client. It seems to occur only
when a large number of files/dirs is created. I can make the archive available to you if
this helps.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Gregory Farnum <gfarnum(a)redhat.com>
Sent: Wednesday, March 22, 2023 4:14 PM
To: Frank Schilder
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
Do you have logs of what the nfs server is doing?
Managed to reproduce it in terms of direct CephFS ops?
On Wed, Mar 22, 2023 at 8:05 AM Frank Schilder
<frans@dtu.dk<mailto:frans@dtu.dk>> wrote:
I have to correct myself. It also fails on an export with "sync" mode. Here is
an strace on the client (strace ln envs/satwindspy/include/ffi.h
mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h):
[...]
stat("mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0x7ffdc5c32820) =
-1 ENOENT (No such file or directory)
lstat("envs/satwindspy/include/ffi.h", {st_mode=S_IFREG|0664, st_size=13934,
...}) = 0
linkat(AT_FDCWD, "envs/satwindspy/include/ffi.h", AT_FDCWD,
"mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0) = -1 EROFS (Read-only
file system)
[...]
write(2, "ln: ", 4ln: ) = 4
write(2, "failed to create hard link 'mamb"..., 80failed to create hard
link 'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h') = 80
[...]
write(2, ": Read-only file system", 23: Read-only file system) = 23
write(2, "\n", 1
) = 1
lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
close(0) = 0
close(1) = 0
close(2) = 0
exit_group(1) = ?
+++ exited with 1 +++
Has anyone advice?
Thanks!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>>
Sent: Wednesday, March 22, 2023 2:44 PM
To: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: [ceph-users] ln: failed to create hard link 'file name': Read-only file
system
Hi all,
on an NFS re-export of a ceph-fs (kernel client) I observe a very strange error. I'm
un-taring a larger package (1.2G) and after some time I get these errors:
ln: failed to create hard link 'file name': Read-only file system
The strange thing is that this seems only temporary. When I used "ln src dst"
for manual testing, the command failed as above. However, after that I tried "ln -v
src dst" and this command created the hard link with exactly the same path arguments.
During the period when the error occurs, I can't see any FS in read-only mode, neither
on the NFS client nor the NFS server. Funny thing is that file creation and write still
works, its only the hard-link creation that fails.
For details, the set-up is:
file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to other server
other server: mount /shares/path as NFS
More precisely, on the file-server:
fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph
defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev 0 0
exports: /shares/nfs/folder -no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP
On the host at DEST-IP:
fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev 0 0
Both, the file server and the client server are virtual machines. The file server is on
Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine is on AlmaLinux 8
(4.18.0-425.13.1.el8_7.x86_64).
When I change the NFS export from "async" to "sync" everything works.
However, that's a rather bad workaround and not a solution. Although this looks like
an NFS issue, I'm afraid it is a problem with hard links and ceph-fs. It looks like a
race with scheduling and executing operations on the ceph-fs kernel mount.
Has anyone seen something like that?
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Best Regards,
Xiubo Li (李秀波)
Email: xiubli@redhat.com/xiubli@ibm.com
Slack: @Xiubo Li
7:52 p.m.
Sorry for late.
No worries.
The ceph qa teuthology test cases have already one
similar test, which
will untar a kernel tarball, but never seen this yet.
I will try this again tomorrow without the NFS client.
Great. In case you would like to use the archive I sent you a link for, please keep it
confidential. It contains files not for publication.
I will collect the log information you asked for.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Xiubo Li <xiubli(a)redhat.com>
Sent: Monday, March 27, 2023 4:15 PM
To: Frank Schilder; Gregory Farnum
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
Frank,
Sorry for late.
On 24/03/2023 01:56, Frank Schilder wrote:
Hi Xiubo and Gregory,
sorry for the slow reply, I did some more debugging and didn't have too much time.
First some questions to collecting logs, but please see also below for reproducing the
issue yourselves.
I can reproduce it reliably but need some input for these:
$ echo "module ceph +p" > /sys/kernel/debug/dynamic_debug/control
This will enable the debug logs in kernel ceph. Then please provide the
message logs.
enabling the kclient debug logs and
How do I do that? I thought the kclient ignores the ceph.conf and I'm not
aware of a mount option to this effect. Is there a "ceph config set ..." setting
I can change for a specific client (by host name/IP) and how exactly?
also the mds
debug logs
I guess here I should set a higher loglevel for the MDS serving this directory (it
is pinned to a single rank) or is it something else?
The issue seems to require a certain load to show up. I created a minimal tar file
mimicking the problem and having 2 directories with a hard link from a file in the first
to a new name in the second directory. This does not cause any problems, so its not that
easy to reproduce.
How you can reproduce it:
As an alternative to my limited skills of pulling logs out, I make the tgz-archive
available to you both. You will receive an e-mail from our one-drive with a download link.
If you un-tar the archive on an NFS client dir that's a re-export of a kclient mount,
after some time you should see the errors showing up.
I can reliably reproduce these errors on our production- as well as on our test cluster.
You should be able to reproduce it too with the tgz file.
Here is a result on our set-up:
- production cluster (executed in a sub-dir conda to make cleanup easy):
$ time tar -xzf ../conda.tgz
tar: mambaforge/pkgs/libstdcxx-ng-9.3.0-h6de172a_18/lib/libstdc++.so.6.0.28: Cannot hard
link to ‘envs/satwindspy/lib/libstdc++.so.6.0.28’: Read-only file system
[...]
tar: mambaforge/pkgs/boost-cpp-1.72.0-h9d3c048_4/lib/libboost_log.so.1.72.0: Cannot hard
link to ‘envs/satwindspy/lib/libboost_log.so.1.72.0’: Read-only file system
^C
real 1m29.008s
user 0m0.612s
sys 0m6.870s
By this time there are already hard links created, so it doesn't fail right away:
$ find -type f -links +1
./mambaforge/pkgs/libev-4.33-h516909a_1/share/man/man3/ev.3
./mambaforge/pkgs/libev-4.33-h516909a_1/include/ev++.h
./mambaforge/pkgs/libev-4.33-h516909a_1/include/ev.h
...
- test cluster (octopus latest stable, 3 OSD hosts with 3 HDD OSDs each, simple
ceph-fs):
# ceph fs status
fs - 2 clients
==
RANK STATE MDS ACTIVITY DNS INOS
0 active tceph-02 Reqs: 0 /s 1807k 1739k
POOL TYPE USED AVAIL
fs-meta1 metadata 18.3G 156G
fs-meta2 data 0 156G
fs-data data 1604G 312G
STANDBY MDS
tceph-01
tceph-03
MDS version: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus
(stable)
Its the new recommended 3-pool layout with fs-data being a 4+2 EC pool.
$ time tar -xzf / ... /conda.tgz
tar: mambaforge/ssl/cacert.pem: Cannot hard link to ‘envs/satwindspy/ssl/cacert.pem’:
Read-only file system
[...]
tar: mambaforge/lib/engines-1.1/padlock.so: Cannot hard link to
‘envs/satwindspy/lib/engines-1.1/padlock.so’: Read-only file system
^C
real 6m23.522s
user 0m3.477s
sys 0m25.792s
Same story here, a large number of hard links has already been created before it starts
failing:
$ find -type f -links +1
./mambaforge/lib/liblzo2.so.2.0.0
...
Looking at the output of find in both cases it also looks a bit non-deterministic when it
starts failing.
It would be great if you can reproduce the issue on a similar test setup using the
archive conda.tgz. If not, I'm happy to collect any type of logs on our test
cluster.
We have now one user who has problems with rsync to an NFS share and it would be really
appreciated if this could be sorted.
The ceph qa teuthology test cases have already one similar test, which
will untar a kernel tarball, but never seen this yet.
I will try this again tomorrow without the NFS client.
Thanks
- Xiubo
Thanks for your help and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Xiubo Li <xiubli(a)redhat.com>
Sent: Thursday, March 23, 2023 2:41 AM
To: Frank Schilder; Gregory Farnum
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
Hi Frank,
Could you reproduce it again by enabling the kclient debug logs and also
the mds debug logs ?
I need to know what exactly has happened in kclient and mds side.
Locally I couldn't reproduce it.
Thanks
- Xiubo
On 22/03/2023 23:27, Frank Schilder wrote:
--
Best Regards,
Xiubo Li (李秀波)
Email: xiubli@redhat.com/xiubli@ibm.com
Slack: @Xiubo Li
Hi Gregory,
thanks for your reply. First a quick update. Here is how I get ln to work after it
failed, there seems no timeout:
$ ln envs/satwindspy/include/ffi.h mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
ln: failed to create hard link
'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h': Read-only file system
$ ls -l envs/satwindspy/include mambaforge/pkgs/libffi-3.3-h58526e2_2
envs/satwindspy/include:
total 7664
-rw-rw-r--. 1 rit rit 959 Mar 5 2021 ares_build.h
[...]
$ ln envs/satwindspy/include/ffi.h mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
After an ls -l on both directories ln works.
To the question: How can I pull out a log from the nfs server? There is nothing in
/var/log/messages.
I can't reproduce it with simple commands on the NFS client. It seems to occur only
when a large number of files/dirs is created. I can make the archive available to you if
this helps.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Gregory Farnum <gfarnum(a)redhat.com>
Sent: Wednesday, March 22, 2023 4:14 PM
To: Frank Schilder
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
Do you have logs of what the nfs server is doing?
Managed to reproduce it in terms of direct CephFS ops?
On Wed, Mar 22, 2023 at 8:05 AM Frank Schilder
<frans@dtu.dk<mailto:frans@dtu.dk>> wrote:
I have to correct myself. It also fails on an export with "sync" mode. Here is
an strace on the client (strace ln envs/satwindspy/include/ffi.h
mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h):
[...]
stat("mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0x7ffdc5c32820) =
-1 ENOENT (No such file or directory)
lstat("envs/satwindspy/include/ffi.h", {st_mode=S_IFREG|0664, st_size=13934,
...}) = 0
linkat(AT_FDCWD, "envs/satwindspy/include/ffi.h", AT_FDCWD,
"mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0) = -1 EROFS (Read-only
file system)
[...]
write(2, "ln: ", 4ln: ) = 4
write(2, "failed to create hard link 'mamb"..., 80failed to create hard
link 'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h') = 80
[...]
write(2, ": Read-only file system", 23: Read-only file system) = 23
write(2, "\n", 1
) = 1
lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
close(0) = 0
close(1) = 0
close(2) = 0
exit_group(1) = ?
+++ exited with 1 +++
Has anyone advice?
Thanks!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>>
Sent: Wednesday, March 22, 2023 2:44 PM
To: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: [ceph-users] ln: failed to create hard link 'file name': Read-only file
system
Hi all,
on an NFS re-export of a ceph-fs (kernel client) I observe a very strange error. I'm
un-taring a larger package (1.2G) and after some time I get these errors:
ln: failed to create hard link 'file name': Read-only file system
The strange thing is that this seems only temporary. When I used "ln src dst"
for manual testing, the command failed as above. However, after that I tried "ln -v
src dst" and this command created the hard link with exactly the same path arguments.
During the period when the error occurs, I can't see any FS in read-only mode, neither
on the NFS client nor the NFS server. Funny thing is that file creation and write still
works, its only the hard-link creation that fails.
For details, the set-up is:
file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to other server
other server: mount /shares/path as NFS
More precisely, on the file-server:
fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph
defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev 0 0
exports: /shares/nfs/folder -no_root_squash,rw,async,mountpoint,no_subtree_check
DEST-IP
On the host at DEST-IP:
fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev 0 0
Both, the file server and the client server are virtual machines. The file server is on
Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine is on AlmaLinux 8
(4.18.0-425.13.1.el8_7.x86_64).
When I change the NFS export from "async" to "sync" everything works.
However, that's a rather bad workaround and not a solution. Although this looks like
an NFS issue, I'm afraid it is a problem with hard links and ceph-fs. It looks like a
race with scheduling and executing operations on the ceph-fs kernel mount.
Has anyone seen something like that?
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Best Regards,
Xiubo Li (李秀波)
Email: xiubli@redhat.com/xiubli@ibm.com
Slack: @Xiubo Li
28 Mar
28 Mar
3:05 a.m.
Dear Xiubo,
I managed to collect logs and uploaded them to:
ceph-post-file: 3d4d1419-a11e-4937-b0b1-bd99234d4e57
By the way, if you run the test with the conda.tgz at the link location, be careful: it
contains a .bashrc file to activate the conda environment. Un-tar it only in a dedicated
location. Unfortunately, this is default with a conda installation. I will remove this
file from the archive tomorrow. Well, I hope the logs contain what you are looking for.
I enabled dmesg debug logs for both, the kclient and nfsd. However, nfsd seems not to log
anything, I see only ceph messages. I interrupted the tar command as soon as the error
showed up for a number of times. There is indeed a change in log messages at the end,
indicating an issue with ceph client caps under high load.
It looks as if instead of waiting for an MDS response the kclient completes a request
prematurely with insufficient caps. I really hope it is possible to fix that.
I will keep the files on the system in case you need FS info for specific inodes.
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: Monday, March 27, 2023 5:22 PM
To: Xiubo Li; Gregory Farnum
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
Sorry for late.
No worries.
The ceph qa teuthology test cases have already one
similar test, which
will untar a kernel tarball, but never seen this yet.
I will try this again tomorrow without the NFS client.
Great. In case you would like to use the archive I sent you a link for, please keep it
confidential. It contains files not for publication.
I will collect the log information you asked for.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
5:08 a.m.
Hi Frank,
Thanks very much for your logs.
I will check it.
- Xiubo
On 28/03/2023 06:35, Frank Schilder wrote:
Dear Xiubo,
I managed to collect logs and uploaded them to:
ceph-post-file: 3d4d1419-a11e-4937-b0b1-bd99234d4e57
By the way, if you run the test with the conda.tgz at the link location, be careful: it
contains a .bashrc file to activate the conda environment. Un-tar it only in a dedicated
location. Unfortunately, this is default with a conda installation. I will remove this
file from the archive tomorrow. Well, I hope the logs contain what you are looking for.
I enabled dmesg debug logs for both, the kclient and nfsd. However, nfsd seems not to log
anything, I see only ceph messages. I interrupted the tar command as soon as the error
showed up for a number of times. There is indeed a change in log messages at the end,
indicating an issue with ceph client caps under high load.
It looks as if instead of waiting for an MDS response the kclient completes a request
prematurely with insufficient caps. I really hope it is possible to fix that.
I will keep the files on the system in case you need FS info for specific inodes.
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: Monday, March 27, 2023 5:22 PM
To: Xiubo Li; Gregory Farnum
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
--
Best Regards,
Xiubo Li (李秀波)
Email: xiubli@redhat.com/xiubli@ibm.com
Slack: @Xiubo Li
Sorry for late.
No worries.
The ceph qa teuthology test cases have already
one similar test, which
will untar a kernel tarball, but never seen this yet.
I will try this again tomorrow without the NFS client.
Great. In case you would
like to use the archive I sent you a link for, please keep it confidential. It contains
files not for publication.
I will collect the log information you asked for.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
26 May
26 May
5:13 p.m.
Update to the list: after extensive debugging by Xiubo on our test cluster, the issue was
identified and fixed. A patch in on its way to distro kernels. The tracker for this case
is: https://tracker.ceph.com/issues/59515
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: Thursday, March 23, 2023 6:56 PM
To: Xiubo Li; Gregory Farnum
Cc: ceph-users(a)ceph.io
Subject: [ceph-users] Re: ln: failed to create hard link 'file name': Read-only
file system
Hi Xiubo and Gregory,
sorry for the slow reply, I did some more debugging and didn't have too much time.
First some questions to collecting logs, but please see also below for reproducing the
issue yourselves.
I can reproduce it reliably but need some input for these:
enabling the kclient debug logs and
How do I do
that? I thought the kclient ignores the ceph.conf and I'm not aware of a mount option
to this effect. Is there a "ceph config set ..." setting I can change for a
specific client (by host name/IP) and how exactly?
also the mds debug logs
I guess here I should
set a higher loglevel for the MDS serving this directory (it is pinned to a single rank)
or is it something else?
The issue seems to require a certain load to show up. I created a minimal tar file
mimicking the problem and having 2 directories with a hard link from a file in the first
to a new name in the second directory. This does not cause any problems, so its not that
easy to reproduce.
How you can reproduce it:
As an alternative to my limited skills of pulling logs out, I make the tgz-archive
available to you both. You will receive an e-mail from our one-drive with a download link.
If you un-tar the archive on an NFS client dir that's a re-export of a kclient mount,
after some time you should see the errors showing up.
I can reliably reproduce these errors on our production- as well as on our test cluster.
You should be able to reproduce it too with the tgz file.
Here is a result on our set-up:
- production cluster (executed in a sub-dir conda to make cleanup easy):
$ time tar -xzf ../conda.tgz
tar: mambaforge/pkgs/libstdcxx-ng-9.3.0-h6de172a_18/lib/libstdc++.so.6.0.28: Cannot hard
link to ‘envs/satwindspy/lib/libstdc++.so.6.0.28’: Read-only file system
[...]
tar: mambaforge/pkgs/boost-cpp-1.72.0-h9d3c048_4/lib/libboost_log.so.1.72.0: Cannot hard
link to ‘envs/satwindspy/lib/libboost_log.so.1.72.0’: Read-only file system
^C
real 1m29.008s
user 0m0.612s
sys 0m6.870s
By this time there are already hard links created, so it doesn't fail right away:
$ find -type f -links +1
./mambaforge/pkgs/libev-4.33-h516909a_1/share/man/man3/ev.3
./mambaforge/pkgs/libev-4.33-h516909a_1/include/ev++.h
./mambaforge/pkgs/libev-4.33-h516909a_1/include/ev.h
...
- test cluster (octopus latest stable, 3 OSD hosts with 3 HDD OSDs each, simple ceph-fs):
# ceph fs status
fs - 2 clients
==
RANK STATE MDS ACTIVITY DNS INOS
0 active tceph-02 Reqs: 0 /s 1807k 1739k
POOL TYPE USED AVAIL
fs-meta1 metadata 18.3G 156G
fs-meta2 data 0 156G
fs-data data 1604G 312G
STANDBY MDS
tceph-01
tceph-03
MDS version: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus
(stable)
Its the new recommended 3-pool layout with fs-data being a 4+2 EC pool.
$ time tar -xzf / ... /conda.tgz
tar: mambaforge/ssl/cacert.pem: Cannot hard link to ‘envs/satwindspy/ssl/cacert.pem’:
Read-only file system
[...]
tar: mambaforge/lib/engines-1.1/padlock.so: Cannot hard link to
‘envs/satwindspy/lib/engines-1.1/padlock.so’: Read-only file system
^C
real 6m23.522s
user 0m3.477s
sys 0m25.792s
Same story here, a large number of hard links has already been created before it starts
failing:
$ find -type f -links +1
./mambaforge/lib/liblzo2.so.2.0.0
...
Looking at the output of find in both cases it also looks a bit non-deterministic when it
starts failing.
It would be great if you can reproduce the issue on a similar test setup using the archive
conda.tgz. If not, I'm happy to collect any type of logs on our test cluster.
We have now one user who has problems with rsync to an NFS share and it would be really
appreciated if this could be sorted.
Thanks for your help and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Xiubo Li <xiubli(a)redhat.com>
Sent: Thursday, March 23, 2023 2:41 AM
To: Frank Schilder; Gregory Farnum
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
Hi Frank,
Could you reproduce it again by enabling the kclient debug logs and also
the mds debug logs ?
I need to know what exactly has happened in kclient and mds side.
Locally I couldn't reproduce it.
Thanks
- Xiubo
On 22/03/2023 23:27, Frank Schilder wrote:
Hi Gregory,
thanks for your reply. First a quick update. Here is how I get ln to work after it
failed, there seems no timeout:
$ ln envs/satwindspy/include/ffi.h mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
ln: failed to create hard link
'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h': Read-only file system
$ ls -l envs/satwindspy/include mambaforge/pkgs/libffi-3.3-h58526e2_2
envs/satwindspy/include:
total 7664
-rw-rw-r--. 1 rit rit 959 Mar 5 2021 ares_build.h
[...]
$ ln envs/satwindspy/include/ffi.h mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
After an ls -l on both directories ln works.
To the question: How can I pull out a log from the nfs server? There is nothing in
/var/log/messages.
I can't reproduce it with simple commands on the NFS client. It seems to occur only
when a large number of files/dirs is created. I can make the archive available to you if
this helps.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Gregory Farnum <gfarnum(a)redhat.com>
Sent: Wednesday, March 22, 2023 4:14 PM
To: Frank Schilder
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: ln: failed to create hard link 'file name':
Read-only file system
Do you have logs of what the nfs server is doing?
Managed to reproduce it in terms of direct CephFS ops?
On Wed, Mar 22, 2023 at 8:05 AM Frank Schilder
<frans@dtu.dk<mailto:frans@dtu.dk>> wrote:
I have to correct myself. It also fails on an export with "sync" mode. Here is
an strace on the client (strace ln envs/satwindspy/include/ffi.h
mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h):
[...]
stat("mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0x7ffdc5c32820) =
-1 ENOENT (No such file or directory)
lstat("envs/satwindspy/include/ffi.h", {st_mode=S_IFREG|0664, st_size=13934,
...}) = 0
linkat(AT_FDCWD, "envs/satwindspy/include/ffi.h", AT_FDCWD,
"mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0) = -1 EROFS (Read-only
file system)
[...]
write(2, "ln: ", 4ln: ) = 4
write(2, "failed to create hard link 'mamb"..., 80failed to create hard
link 'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h') = 80
[...]
write(2, ": Read-only file system", 23: Read-only file system) = 23
write(2, "\n", 1
) = 1
lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
close(0) = 0
close(1) = 0
close(2) = 0
exit_group(1) = ?
+++ exited with 1 +++
Has anyone advice?
Thanks!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>>
Sent: Wednesday, March 22, 2023 2:44 PM
To: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: [ceph-users] ln: failed to create hard link 'file name': Read-only file
system
Hi all,
on an NFS re-export of a ceph-fs (kernel client) I observe a very strange error. I'm
un-taring a larger package (1.2G) and after some time I get these errors:
ln: failed to create hard link 'file name': Read-only file system
The strange thing is that this seems only temporary. When I used "ln src dst"
for manual testing, the command failed as above. However, after that I tried "ln -v
src dst" and this command created the hard link with exactly the same path arguments.
During the period when the error occurs, I can't see any FS in read-only mode, neither
on the NFS client nor the NFS server. Funny thing is that file creation and write still
works, its only the hard-link creation that fails.
For details, the set-up is:
file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to other server
other server: mount /shares/path as NFS
More precisely, on the file-server:
fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph
defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev 0 0
exports: /shares/nfs/folder -no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP
On the host at DEST-IP:
fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev 0 0
Both, the file server and the client server are virtual machines. The file server is on
Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine is on AlmaLinux 8
(4.18.0-425.13.1.el8_7.x86_64).
When I change the NFS export from "async" to "sync" everything works.
However, that's a rather bad workaround and not a solution. Although this looks like
an NFS issue, I'm afraid it is a problem with hard links and ceph-fs. It looks like a
race with scheduling and executing operations on the ceph-fs kernel mount.
Has anyone seen something like that?
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Best Regards,
Xiubo Li (李秀波)
Email: xiubli@redhat.com/xiubli@ibm.com
Slack: @Xiubo Li
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
330
days inactive
395
days old
13 comments
4 participants
participants (4)
-
Christophe BAILLON -
Frank Schilder -
Gregory Farnum -
Xiubo Li