Hi, I just tried to deploy an NFS service on a freshly installed 20.2.1 cluster and it fails. ceph nfs cluster create nfs01 "label:nfs01" There is exactly one host with the label nfs01. There are some entries in the log showing errors but the last two are - Cannot acquire credentials for principal nfs - Cannot register NFS V3 on TCP /var/lib/ceph/2ac9b090-3363-11f1-bd6a-bc2411d682b7/nfs.nfs01.0.0.rgw01.hpfeuz/etc/ganesha/ganesha.conf contained "Protocols 3, 4;" in the NFS_CORE_PARAM section. After removing the 3 the container starts. NFSv3 was added in commit 1e69ebc6ceaf in February 2024. It was not contained in Ceph squid. Apr 09 14:23:18 HOST systemd[1]: Starting ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7@nfs.nfs01.0.0.HOST.hpfeuz.service - Ceph nfs.nfs01.0.0.HOST.hpfeuz for 2ac9b090-3363-11f1-bd6a-bc2411d682b7... Apr 09 14:23:18 HOST podman[31327]: 2026-04-09 14:23:18.942654365 +0200 CEST m=+0.048118777 container create c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_REF=tentacle, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260331, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.43.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) Apr 09 14:23:19 HOST podman[31327]: 2026-04-09 14:23:19.000318083 +0200 CEST m=+0.105782538 container init c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., io.buildah.version=1.43.0, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_REF=tentacle, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260331) Apr 09 14:23:19 HOST podman[31327]: 2026-04-09 14:23:19.004217805 +0200 CEST m=+0.109682235 container start c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_REF=tentacle, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.vendor=CentOS, ceph=True, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260331, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, io.buildah.version=1.43.0, org.label-schema.license=GPLv2) Apr 09 14:23:19 HOST bash[31327]: c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 Apr 09 14:23:19 HOST podman[31327]: 2026-04-09 14:23:18.922220554 +0200 CEST m=+0.027684991 image pull fb63cba66eea5516fbd9a1ec035c265abdbc1a1a8fc2dc0b51057557e49ca28c quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337 Apr 09 14:23:19 HOST systemd[1]: Started ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7@nfs.nfs01.0.0.HOST.hpfeuz.service - Ceph nfs.nfs01.0.0.HOST.hpfeuz for 2ac9b090-3363-11f1-bd6a-bc2411d682b7. Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] init_logging :LOG :NULL :LOG: Setting log level for all components to NIV_EVENT Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.9 Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] monitoring_init :NFS STARTUP :EVENT :Init monitoring at 0.0.0.0:9587 Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit on FDs to 1048576. Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] rados_kv_traverse :CLIENT ID :EVENT :Failed to lst kv ret=-2 Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] rados_cluster_read_clids :CLIENT ID :EVENT :Failed to traverse recovery db: -2 Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend Apr 09 14:23:19 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:19 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] rados_cluster_end_grace :CLIENT ID :EVENT :Failed to remove rec-0000000000000001:0: -2 Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!! Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:9): Unknown parameter (allow_set_io_flusher_fail) Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:28): Unknown block (RADOS_URLS) Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:33): Unknown block (RGW) Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory) Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs Apr 09 14:23:22 HOST ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz[31339]: 09/04/2026 12:23:22 : epoch 69d79a37 : HOST : ganesha.nfsd-2[main] __Register_program :DISP :MAJ :Cannot register NFS V3 on TCP Apr 09 14:23:22 HOST podman[31583]: 2026-04-09 14:23:22.269193428 +0200 CEST m=+0.027823817 container died c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, io.buildah.version=1.43.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=tentacle, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., org.label-schema.build-date=20260331, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>) Apr 09 14:23:22 HOST podman[31583]: 2026-04-09 14:23:22.325861901 +0200 CEST m=+0.084492274 container remove c2b0e96637297b20d6bd39bf1eb68fc70b0fb5635dc72c9b887eeb82cd2bac77 (image=quay.io/ceph/ceph@sha256:0bae386bc859cd9a05b804d1ca16cca8853a64f90809044e2bf43095419dc337, name=ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7-nfs-nfs01-0-0-HOST-hpfeuz, ceph=True, org.label-schema.vendor=CentOS, CEPH_REF=tentacle, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfs..., OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_SHA1=6a49aff47758778a5f5951e731d437c317f72fb2, org.label-schema.build-date=20260331, io.buildah.version=1.43.0, org.opencontainers.image.authors=Ceph Release Team <ceph-maintainers@ceph.io>, CEPH_GIT_REPO=https://github.com/ceph/ceph.git, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) Apr 09 14:23:22 HOST systemd[1]: ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7@nfs.nfs01.0.0.HOST.hpfeuz.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Apr 09 14:23:22 HOST systemd[1]: ceph-2ac9b090-3363-11f1-bd6a-bc2411d682b7@nfs.nfs01.0.0.HOST.hpfeuz.service: Failed with result 'exit-code'. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin