Hi all, I have a Ceph Cluster using squid 19.2.2. I'm using CephFS storage with kernel mounts on clients and it works fine but for some older computers (still CentOS8 and one CentOS7 because it is running a commercial application) I need to use NFS so I've deployed Ganesha. But a few minutes after the first NFS client mount the volume (Just testing from the first client) and uses it, the Ceph NFS service falls in error on one node (not always the same node) and I have to reboot it. # ceph orch ps --daemon-type nfs NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID nfs.whitaker-nfs.0.2.whitaker04-ceph.itzyqf whitaker04-ceph *:12049 running (49m) 5m ago 3h 26.3M - 5.9 4892a7ef541b 42fd4ab7bba9 nfs.whitaker-nfs.1.0.whitaker05-ceph.vocxmr whitaker05-ceph *:12049 running (63m) 5m ago 23h 85.7M - 5.9 4892a7ef541b ab1c06be2ce7 nfs.whitaker-nfs.2.0.whitaker01-ceph.rhwuth whitaker01-ceph *:12049 running (33m) 5m ago 23h 24.6M - 5.9 4892a7ef541b eb93963f05cd nfs.whitaker-nfs.3.0.whitaker03-ceph.wqfnnq whitaker03-ceph *:12049 error 5m ago 23h - - <unknown> <unknown> <unknown> nfs.whitaker-nfs.4.0.whitaker02-ceph.wjynvn whitaker02-ceph *:12049 running (49m) 4m ago 3h 28.3M - 5.9 4892a7ef541b 5cb71f9d8b89 and the client freeze on the mount point. Any idea about this or how to track the problem ? Thanks Patrick ---------------------------------------------- Details of my NFS CEPH cluster, building commands and status: ---------------------------------------------- # *ceph nfs cluster create whitaker-nfs \ whitaker01-ceph,whitaker02-ceph,whitaker03-ceph,whitaker04-ceph,whitaker05-ceph \ --ingress --virtual_ip 172.30.1.102/24 --ingress-mode haproxy-protocol* *# ceph nfs cluster info whitaker-nfs* { "whitaker-nfs": { "backend": [ { "hostname": "whitaker01-ceph", "ip": "172.30.1.97", "port": 12049 }, { "hostname": "whitaker02-ceph", "ip": "172.30.1.98", "port": 12049 }, { "hostname": "whitaker03-ceph", "ip": "172.30.1.99", "port": 12049 }, { "hostname": "whitaker04-ceph", "ip": "172.30.1.100", "port": 12049 }, { "hostname": "whitaker05-ceph", "ip": "172.30.1.101", "port": 12049 } ], "monitor_port": 9049, "port": 2049, "virtual_ip": "172.30.1.102" } } *# ceph nfs export create cephfs \ --cluster-id whitaker-nfs \ --pseudo-path /data \ --fsname data \ --client_addr 172.30.1.0/24* *# ceph nfs export ls whitaker-nfs --detailed* [ { "access_type": "none", "clients": [ { "access_type": "rw", "addresses": [ "172.30.1.0/24" ], "squash": "none" } ], "cluster_id": "whitaker-nfs", "export_id": 1, "fsal": { "cmount_path": "/", "fs_name": "data", "name": "CEPH", "user_id": "nfs.whitaker-nfs.data.79da3503" }, "path": "/", "protocols": [ 4 ], "pseudo": "/data", "security_label": true, "squash": "none", "transports": [ "TCP" ] } ]
Try setenforce 0 as a test. If stable, review ausearch -m avc and adjust SELinux policy.
Hi Georgi, thanks for your suggestion. However setting "setenforce 0" on the nodes do not change the situation. The ganesha.nfsd processes still crash randomly with the same error if I set ingress-mode to "haproxy-protocol" in SQUID version of Ceph: [5543824.766357] ganesha.nfsd[2601171]: segfault at 50 ip 00007ff64a4ed32e sp 00007ff5b37fd210 error 4 in libntirpc.so.5.8[7ff64a4d2000+2c000] likely on CPU 12 (core 12, socket 0) [5543824.768336] Code: 47 20 66 41 89 86 f2 00 00 00 41 bf 01 00 00 00 b9 40 00 00 00 e9 af fd ff ff 66 90 48 8b 85 f8 00 00 00 48 8b 40 08 4c 8b 28 <45> 8b 65 50 49 8b 75 68 41 8b be 28 02 00 00 b9 40 00 00 00 e8 29 The libntirpc package (providing libntirpc.so.5.8) only exist in the ceph containerized environment, not in the host list packages. Patrick Le 26/09/2025 à 17:28, Georgi Steffenhagen a écrit :
Try setenforce 0 as a test. If stable, review ausearch -m avc and adjust SELinux policy. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Patrick, we gave up on using the HAProxy option, see this thread from July 2025: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/DWFPRZC6X5C...
Thank Nigel, I was not aware of this bug report 🙁. At this time, with Squid 19.2.2 ingress mode with haproxy-standard looks stable, only the haproxy-protocol approach fails. No idea about 19.2.3 if it resolves the bug. I just need NFS for a small number of hosts, still in an old linux version at this time. May be haproxy-standard could be enough in my environment. I saw the I/O performances are quite bad with ganesha, significantly slower than my old standalone NFS server. Moving to kernel mounts with cephfs after updating the clients will be the best way in the next months. But I hope developers will find and solve this bad bug. Patrick Le 01/10/2025 à 00:22, Nigel Williams a écrit :
we gave up on using the HAProxy option, see this thread from July 2025
On Wed, 1 Oct 2025 at 17:35, Patrick Begou < Patrick.Begou@univ-grenoble-alpes.fr> wrote:
... I saw the I/O performances are quite bad with ganesha, significantly slower than my old standalone NFS server. Moving to kernel mounts with cephfs after updating the clients will be the best way in the next months.
We recently spun-up a new cluster on current hardware (in our case Dell R760XD2, spinners and rear bay with E3.S NVMe) using Squid and are seeing much better Ganesha performance than we have experienced in the past. Good enough that we expect we can continue with Ganesha and not use kernel NFS. cheers, nigel.
Hi Nigel, you can use Kernel NFS with CEPH ? I'm a new user with Ganesha, and I do not understand what this is ? My comparison is based on - a DELL R740 server, 16 HDDs, PERC H730P, 2 raid6 stripped with 8HDD each, NFS v4, 10Gb ethernet, 64GB RAM - 5 HPE Proliant DL345, 4HDDs each (so 20HDD), Squid 19.2.2, EC 5+3 setup (2 chuncks/node), Ganesha, NFSv4, 25Gb ethernet (only one network), 128GB RAM each. - client has 25Gb ethernet too. Test is 4 processes writing or reading 4 remotes files of 4GB 10 time (so write 160GB then read 160GB) - NFS server test: write avg 287Mo/s, read avg 802Mo/s - Ganesha test: write avg *225*Mo/s, read avg*340*Mo/s - CephFS (kernel) test: write avg 767Mo/s, read avg 1079Mo/s It is the first step for building a Ceph Cluster so 5 nodes and few disks at this time in the cluster. HDD is because we need capacitive storage. Data access profile is few access (codes are not writing or reading frequently) but intensive access (tens to hundreds parallel processes reading ou writing synchronously). Patrick Le 02/10/2025 à 00:02, Nigel Williams a écrit :
On Wed, 1 Oct 2025 at 17:35, Patrick Begou < Patrick.Begou@univ-grenoble-alpes.fr> wrote:
... I saw the I/O performances are quite bad with ganesha, significantly slower than my old standalone NFS server. Moving to kernel mounts with cephfs after updating the clients will be the best way in the next months.
We recently spun-up a new cluster on current hardware (in our case Dell R760XD2, spinners and rear bay with E3.S NVMe) using Squid and are seeing much better Ganesha performance than we have experienced in the past. Good enough that we expect we can continue with Ganesha and not use kernel NFS.
cheers, nigel. _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
participants (3)
-
Georgi Steffenhagen
-
Nigel Williams
-
Patrick Begou