Write issues on CephFS mounted with root_squash
Dear Ceph users, I'm trying to export a CephFS with the root_squash option. This is the client configuration: client.wizardfs_rootsquash key: XXXXXXXXXXXX caps: [mds] allow rw fsname=wizardfs root_squash caps: [mon] allow r fsname=wizardfs caps: [osd] allow rw tag cephfs data=wizardfs I can mount it flawlessly on several machines using the kernel driver, but when a machine writes on it then the content seems fine from the writing machine but it's not actually written on disk since other machines just see an empty file: [12:43 mori@stryke ~]$ echo test > /wizard/ceph/software/el9/test [12:43 mori@stryke ~]$ ll /wizard/ceph/software/el9/test -rw-r--r-- 1 mori wizard 5 mag 15 12:43 /wizard/ceph/software/el9/test [12:43 mori@stryke ~]$ cat /wizard/ceph/software/el9/test test [12:43 mori@stryke ~]$ [mori@fili ~]$ ll /wizard/ceph/software/el9/test -rw-r--r--. 1 mori 1014 0 May 15 06:43 /wizard/ceph/software/el9/test [mori@fili ~]$ cat /wizard/ceph/software/el9/test [mori@fili ~]$ Unmounting and then remounting on "stryke" the file is seen as empty, so I guess that the content shown just after the write is only a cache effect and nothing is effectively written on disk. I checked the posix permissions on the folder and I have rw rights from both the machines. All of the above using Ceph 18.2.2 on the cluster (deployed with cephadm) and both the machines. Machine "fili" has kernel 5.14.0 while "stryke" has 6.8.9. The same issue happens consistently also in the reverse direction (writing from "fili" and reading from "stryke"), and also with other machines. Removing the squash_root option the problem vanishes. I don't know what might
Hey Nicola, Try mounting cephfs with fuse instead of kernel, we have seen before sometimes the kernel mount does not properly support that option but the fuse mount does. Regards, Bailey
-----Original Message----- From: Nicola Mori <mori@fi.infn.it> Sent: May 15, 2024 7:55 AM To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Write issues on CephFS mounted with root_squash
Dear Ceph users,
I'm trying to export a CephFS with the root_squash option. This is the client configuration:
client.wizardfs_rootsquash key: XXXXXXXXXXXX caps: [mds] allow rw fsname=wizardfs root_squash caps: [mon] allow r fsname=wizardfs caps: [osd] allow rw tag cephfs data=wizardfs
I can mount it flawlessly on several machines using the kernel driver, but when a machine writes on it then the content seems fine from the writing machine but it's not actually written on disk since other machines just see an empty file:
[12:43 mori@stryke ~]$ echo test > /wizard/ceph/software/el9/test [12:43 mori@stryke ~]$ ll /wizard/ceph/software/el9/test -rw-r--r-- 1 mori wizard 5 mag 15 12:43 /wizard/ceph/software/el9/test [12:43 mori@stryke ~]$ cat /wizard/ceph/software/el9/test test [12:43 mori@stryke ~]$
[mori@fili ~]$ ll /wizard/ceph/software/el9/test -rw-r--r--. 1 mori 1014 0 May 15 06:43 /wizard/ceph/software/el9/test [mori@fili ~]$ cat /wizard/ceph/software/el9/test [mori@fili ~]$
Unmounting and then remounting on "stryke" the file is seen as empty, so I guess that the content shown just after the write is only a cache effect and nothing is effectively written on disk. I checked the posix permissions on the folder and I have rw rights from both the machines.
All of the above using Ceph 18.2.2 on the cluster (deployed with cephadm) and both the machines. Machine "fili" has kernel 5.14.0 while "stryke" has 6.8.9. The same issue happens consistently also in the reverse direction (writing from "fili" and reading from "stryke"), and also with other machines.
Removing the squash_root option the problem vanishes.
I don't know what might
Thank you Bailey, I'll give it a try ASAP. By the way, is this issue with the kernel driver something that will be fixed at a given point? If I'm correct the kernel driver has better performance than FUSE so I'd like to use it. Cheers, Nicola
Hi, We have the same issue. It seems to come from this bug : https://access.redhat.com/solutions/6982902 We had to disable root_squash, which of course is a huge issue... Cheers, Fabien On 5/15/24 12:54, Nicola Mori wrote:
Dear Ceph users,
I'm trying to export a CephFS with the root_squash option. This is the client configuration:
client.wizardfs_rootsquash key: XXXXXXXXXXXX caps: [mds] allow rw fsname=wizardfs root_squash caps: [mon] allow r fsname=wizardfs caps: [osd] allow rw tag cephfs data=wizardfs
I can mount it flawlessly on several machines using the kernel driver, but when a machine writes on it then the content seems fine from the writing machine but it's not actually written on disk since other machines just see an empty file:
[12:43 mori@stryke ~]$ echo test > /wizard/ceph/software/el9/test [12:43 mori@stryke ~]$ ll /wizard/ceph/software/el9/test -rw-r--r-- 1 mori wizard 5 mag 15 12:43 /wizard/ceph/software/el9/test [12:43 mori@stryke ~]$ cat /wizard/ceph/software/el9/test test [12:43 mori@stryke ~]$
[mori@fili ~]$ ll /wizard/ceph/software/el9/test -rw-r--r--. 1 mori 1014 0 May 15 06:43 /wizard/ceph/software/el9/test [mori@fili ~]$ cat /wizard/ceph/software/el9/test [mori@fili ~]$
Unmounting and then remounting on "stryke" the file is seen as empty, so I guess that the content shown just after the write is only a cache effect and nothing is effectively written on disk. I checked the posix permissions on the folder and I have rw rights from both the machines.
All of the above using Ceph 18.2.2 on the cluster (deployed with cephadm) and both the machines. Machine "fili" has kernel 5.14.0 while "stryke" has 6.8.9. The same issue happens consistently also in the reverse direction (writing from "fili" and reading from "stryke"), and also with other machines.
Removing the squash_root option the problem vanishes.
I don't know what might
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Nicola, Yes, this issue is already fixed in main [1] and the quincy backport is still pending to be merged. Hopefully will be available in the next Quincy release. [1] https://github.com/ceph/ceph/pull/48027 [2] https://github.com/ceph/ceph/pull/54469 Thanks and Regards, Kotresh H R On Wed, May 15, 2024 at 7:51 PM Fabien Sirjean <fsirjean@eddie.fdn.fr> wrote:
Hi,
We have the same issue. It seems to come from this bug : https://access.redhat.com/solutions/6982902
We had to disable root_squash, which of course is a huge issue...
Cheers,
Fabien
On 5/15/24 12:54, Nicola Mori wrote:
Dear Ceph users,
I'm trying to export a CephFS with the root_squash option. This is the client configuration:
client.wizardfs_rootsquash key: XXXXXXXXXXXX caps: [mds] allow rw fsname=wizardfs root_squash caps: [mon] allow r fsname=wizardfs caps: [osd] allow rw tag cephfs data=wizardfs
I can mount it flawlessly on several machines using the kernel driver, but when a machine writes on it then the content seems fine from the writing machine but it's not actually written on disk since other machines just see an empty file:
[12:43 mori@stryke ~]$ echo test > /wizard/ceph/software/el9/test [12:43 mori@stryke ~]$ ll /wizard/ceph/software/el9/test -rw-r--r-- 1 mori wizard 5 mag 15 12:43 /wizard/ceph/software/el9/test [12:43 mori@stryke ~]$ cat /wizard/ceph/software/el9/test test [12:43 mori@stryke ~]$
[mori@fili ~]$ ll /wizard/ceph/software/el9/test -rw-r--r--. 1 mori 1014 0 May 15 06:43 /wizard/ceph/software/el9/test [mori@fili ~]$ cat /wizard/ceph/software/el9/test [mori@fili ~]$
Unmounting and then remounting on "stryke" the file is seen as empty, so I guess that the content shown just after the write is only a cache effect and nothing is effectively written on disk. I checked the posix permissions on the folder and I have rw rights from both the machines.
All of the above using Ceph 18.2.2 on the cluster (deployed with cephadm) and both the machines. Machine "fili" has kernel 5.14.0 while "stryke" has 6.8.9. The same issue happens consistently also in the reverse direction (writing from "fili" and reading from "stryke"), and also with other machines.
Removing the squash_root option the problem vanishes.
I don't know what might
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you Kotresh! My cluster is currently on Reef 18.2.2, which should be the current version and which is affected. Will the fix be included in the next Reef release? Cheers, Nicola
On Fri, May 17, 2024 at 11:52 AM Nicola Mori <mori@fi.infn.it> wrote:
Thank you Kotresh! My cluster is currently on Reef 18.2.2, which should be the current version and which is affected. Will the fix be included in the next Reef release?
Yes, it's already merged to the reef branch, and should be available in the next reef release. Please look at https://tracker.ceph.com/issues/62952
Cheers,
Nicola
Hi, On 5/17/24 08:51, Kotresh Hiremath Ravishankar wrote:
Yes, it's already merged to the reef branch, and should be available in the next reef release. Please look at https://tracker.ceph.com/issues/62952
This is great news! Many thanks to all involved. F.
participants (4)
-
Bailey Allison
-
Fabien Sirjean
-
Kotresh Hiremath Ravishankar
-
Nicola Mori