Re: [Ceph tentacle 20.2.2] problem with nfs ganesha and nfsv3
Hi, look at this thread [0] and the corresponding links provided in there [1], [2]. Regards, Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/LG2EGDZMIWT... [1] https://tracker.ceph.com/issues/74307 [2] https://tracker.ceph.com/issues/74191 Zitat von wodel youchi via ceph-users <ceph-users@ceph.io>:
Hi,
I'm using tentacle 20.2.2 with podman on Rocky Linux 9. I'm trying to configure nfs-ganesha, but the nfs-tentacle containers won't stay up. I've 3 nodes, all services are collocated.
# ceph nfs cluster create tentacle-nfs "srvceph01,srvceph02,srvceph03"
--ingress --virtual_ip 10.200.24.89 --ingress-mode haproxy-standard --port 12049
When I create the nfs cluster (with HA, haproxy + keepalived). 3 nfs-tentacle-nfs containers are created, one on each node, but they do not stay alive, they start then disappear, again and again, and in the end just one is created, but it does not really work, HAProxy shows that all backends are down.
What I've got from the vanished containers is that the nfs service is looking for rpc-bind which does not exist because ganesha's config file enables both nfs v3 and v4.
# ceph orch ps --service_name nfs.tentacle-nfs
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 unknown 5m ago 6m - - <unknown> <unknown> <unknown> nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 running (5m) 5m ago 6m 15.0M - 5.9 c22719cb0d76 dcb1eab045cd nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 running (4m) 4m ago 6m 14.6M - 5.9 c22719cb0d76 b11d32f4b0cc
# ceph orch ps
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID ... haproxy.nfs.tentacle-nfs.srvceph01.uvueop srvceph01 *:12049,19049 running (6m) 4m ago 6m 11.5M - 2.3.17-d1c9119 e85424b0d443 d4a840202ae0 haproxy.nfs.tentacle-nfs.srvceph02.xujdir srvceph02 *:12049,19049 running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 e85424b0d443 747cf2052cd5 haproxy.nfs.tentacle-nfs.srvceph03.snapkl srvceph03 *:12049,19049 running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 e85424b0d443 3c17cb2a4757 keepalived.nfs.tentacle-nfs.srvceph01.phscfe srvceph01 running (6m) 4m ago 6m 19.9M - 2.2.4 4a3a1ff181d9 d053e0b91210 keepalived.nfs.tentacle-nfs.srvceph02.ubwgoo srvceph02 running (6m) 5m ago 6m 20.1M - 2.2.4 4a3a1ff181d9 6ad75ccccaa0 keepalived.nfs.tentacle-nfs.srvceph03.phbgis srvceph03 running (6m) 5m ago 6m 20.1M - 2.2.4 4a3a1ff181d9 9efebd8d5ba5 mds.cephfs.srvceph01.tyrask srvceph01 running (17m) 4m ago 5d 32.4M - 20.2.2 c22719cb0d76 2f89b9d86343 mds.cephfs.srvceph02.ebyvdg srvceph02 running (17m) 5m ago 5d 33.1M - 20.2.2 c22719cb0d76 5afe19ec45b8 mds.cephfs.srvceph03.miznhi srvceph03 running (17m) 5m ago 5d 146M - 20.2.2 c22719cb0d76 6ee9a906d887 ... nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 unknown 5m ago 7m - - <unknown> <unknown> <unknown> nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 running (5m) 5m ago 7m 15.0M - 5.9 c22719cb0d76 dcb1eab045cd nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 running (4m) 4m ago 7m 14.6M - 5.9 c22719cb0d76 b11d32f4b0cc
...
# podman logs nfs-tentacle-container
04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45
After
# ceph orch restart nfs.tentacle-nfs
I got this
# podman logs nfs-tentacle-container
04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] init_logging :LOG :NULL :LOG: Setting log level for all components to NIV_EVENT 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.9 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] monitoring_init :NFS STARTUP :EVENT :Init monitoring at 0.0.0.0:9587 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit on FDs to 1048576. 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=0 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!! 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:9): Unknown parameter (allow_set_io_flusher_fail) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:28): Unknown block (RADOS_URLS) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:33): Unknown block (RGW) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs *04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] __Register_program :DISP :MAJ :Cannot register NFS V3 on TCP <--------------------------*
podman exec -it f2e2c668e0e3 bash
[root@srvceph01 /]# cat etc/ganesha/ganesha.conf # This file is generated by cephadm. NFS_CORE_PARAM { Enable_NLM = false; Enable_RQUOTA = false; * Protocols = 3, 4; <---------------------------------------------------* mount_path_pseudo = true; Enable_UDP = false; NFS_Port = 22049; allow_set_io_flusher_fail = true; Monitoring_Port = 9587; } NFSv4 { Delegations = false; RecoveryBackend = "rados_cluster"; Minor_Versions = 1, 2; Server_Scope = "bcf27a48-8a8d-11f1-bdd3-fc607038078a-tentacle-nfs"; IdmapConf = "/etc/ganesha/idmap.conf"; } RADOS_KV { UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; nodeid = 2; pool = ".nfs"; namespace = "tentacle-nfs"; } RADOS_URLS { UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; watch_url = "rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs"; } RGW { cluster = "ceph"; name = "client.nfs.tentacle-nfs.2.0.srvceph01.ahcayv-rgw"; } %url rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs
I am not sure if this is the cause of the problem, so I thought, why not enable nfsv3 even if I'm not going to use it? but... Ceph's documentation mentions (tentacle version selected) : https://docs.ceph.com/en/tentacle/mgr/nfs/#create-nfs-ganesha-cluster Create NFS Ganesha Cluster
ceph nfs cluster create <cluster_id> .... [--enable-nfsv3]
The --enable-nfsv3 is not recognized, I've tried to use it to enable nfsv3 (as a workaround), but it is not recognized.
I've also tried to use this from https://docs.ceph.com/en/tentacle/cephadm/services/nfs/#cephadm-ha-nfs :
Service Specification
service_type: nfsservice_id: mynfsplacement: hosts: - host1 - host2spec: port: 12345 monitoring_port: 567 enable_nfsv3: true <------------------- not recognized
The enable_nfsv3 is not recognized.
If this is the cause of the problem, then I'm stuck, I don't know how to disable nfsv3 on ganesha nor how to enable nfsv3 with cephadm !!!!!
Regards. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (1)
-
Eugen Block