- ceph-users - lists.ceph.io

Affects of high pg count on OSD peering
by Elias Carter 05 Aug '26

05 Aug '26

We run Ceph clusters with high PG counts (300 PGs/OSD, 32k total PGs and upwards). Something that I have noticed is that CRUSH changes (marking an OSD as out, or a new OSD joining the cluster) can cause many hundreds of PGs to remap and peer. During this time we get PG_AVAILABILITY health check warnings as the OSDs need to peer with their new PGs before they can serve IOs. RADOS tail latency suffers because IOs sent to a peering PG are blocked until peering completes. I believe that the more PGs we have, the more PGs need to peer, and the longer IOs are blocked for the pool. Does anyone have suggestions on how to deal with PG peering blocking IOs? I was thinking about gradually re-weighting OSDs instead of marking them out or in, but that can result in multiple redundant data moves. Elias

1 0

v20.2.3 Tentacle released
by Yuri Weinstein 05 Aug '26

05 Aug '26

This is the third minor release in the Tentacle series. We recommend that all users update to this release. Notable Changes --------------- MDS (Metadata Server) --------------------- * Fixed a crash and shutdown hang that could occur when ephemeral pins were active and max_mds was set to 0. * Session reclaim could miss blocklisting an old session; this is now handled correctly. * Fixed a case where scrub was unable to identify damage from an irreparable hard link. OSD (Object Storage Daemon) ---------------------------- * BlueStore: Fixed a BlueFS WAL envelope-mode bug that caused write buffer misalignment. * Erasure Coding: Fixed truncate+write planning for EC shard sizes. * Fixed a condition that prevented rolling forward of PG log entries. * Prevented OSDMap::check_health() from asserting when new OSDs are found in a subtree. * Scrub: "repairing" scrubs are now allowed at all times. RGW (RADOS Gateway) -------------------- * Added the ssl_ciphersuites option for the Beast front-end to support TLS 1.3 cipher customization. * Fixed PutObject's canned_acl comparison bug affecting BlockPublicAcls. * Fixed crashes occurring on realm reload. * Implemented CopyObject support for encrypted objects. * Added mTLS client-certificate authentication support for Kafka notifications. * Fixed removal of delete markers in lifecycle processing. RADOS / librados / neorados ---------------------------- * neorados: Fixed an infinite trim loop on empty data log shards. * Fixed cancellation-slot cleanup in librados/asio's associated executor. Dashboard ----------------- * Added a hardware-monitoring dashboard backed by node-proxy metrics. * Fixed a bind-address regression caused by CherryPy isolation. * Added support for adding hosts via CSV upload. * NVMeoF: multiple CLI and UI backports (EC pool support, IO statistics, role management). * Object: The global RGW Roles tab has been removed. Role management is now scoped under RGW Accounts, where roles can be listed, created, edited, and deleted for a selected account. ceph-volume ----------- * Reworked OSD mapper lifecycle handling (LVM + raw) for activate. * Improved detection of rotational media under dm-crypt to bypass the workqueue correctly. * Fixed OSD re-deployment issues with disk-selection filters and DB devices. * Added retry handling for lvs when it returns an empty result or a "devices file is missing" error. NVMe-oF ------- * Added configurable delayed failback (default 0, no delay) to accommodate initiators that recover paths with a delay. * Gateways in the DELETING state now ignore beacons and send empty maps instead of processing them. * rbd_with_crc32c is now enabled by default via cephadm. * Introduced a new NVMeoF mgr module. The module automatically creates the ".nvmeof" metadata pool if it does not already exist. Gateway-group state files are stored in this pool unless the user specifies another pool. The module is enabled automatically on fresh installs; on upgrades from an older cluster, it must be enabled manually. RBD --- * Fixed a use-after-free bug releasing object map locks during deep copy. * Fixed memory leaks in PWL discard operations. * Obsolete primary mirror snapshots are now pruned after relocation. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v20-2-3-tentacle-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.2.3.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 06c2f9c35b67055a8a6fb99d1be236b3c4832ace

1 0

Re: Using CephFS for encrypted Kubernetes volumes
by Stefan Möding 05 Aug '26

05 Aug '26

Hi, Eugen Block via ceph-users <ceph-users(a)ceph.io> writes: > I haven't tried it myself yet so I don't really have an answer. But > I noticed that your OSD caps may be incomplete wrt x permissions: >> osd_caps: 'allow rwx namespace=k8s-test tag cephfs metadata=root, >> allow rw namespace=k8s-test tag cephfs data=root' > The second allow statement doesn't have an "x". Maybe that's the issue > why the workaround fails? We tried that, but the error remains. We now consider to use the following capabilities in combination to creating a dedicated data pool for each tenant (K8s cluster): mon_caps: 'allow r fsname=root' mgr_caps: 'allow rw' osd_caps: 'allow rwx pool=cephfs.meta, allow rw pool=cephfs.data, allow rw pool=cephfs.k8s-test' mds_caps: 'allow r fsname=root path=/volumes, allow rws fsname=root path=/volumes/k8s-test' CephFS is created with cephfs.meta as metadata pool and cephfs.data as default data pool while cephfs.k8s-test is the pool for this tenant. What are the implications to share the metadata/default data pools and have no namespace? Is the MDS capability sufficient to separate tenants or can everyone access the metadata objects of everyone else? -- Stefan

1 0

Re: [Ceph tentacle 20.2.2] problem with nfs ganesha and nfsv3
by wodel youchi 05 Aug '26

05 Aug '26

Hi, Thanks, to remove nfsv3 from ganesha's config, should I use : ceph nfs cluster config set <cluster_id> -i <config_file> with for example : vim <config_file> NFS_CORE_PARAM { * Protocols = 4; } Then # ceph orch restart nfs.tentacle-nfs Regards. Le mar. 4 août 2026 à 11:33, Eugen Block via ceph-users <ceph-users(a)ceph.io> a écrit : > Hi, > > look at this thread [0] and the corresponding links provided in there > [1], [2]. > > Regards, > Eugen > > [0] > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/LG2EGDZMIW… > [1] https://tracker.ceph.com/issues/74307 > [2] https://tracker.ceph.com/issues/74191 > > Zitat von wodel youchi via ceph-users <ceph-users(a)ceph.io>: > > > Hi, > > > > I'm using tentacle 20.2.2 with podman on Rocky Linux 9. > > I'm trying to configure nfs-ganesha, but the nfs-tentacle containers > won't > > stay up. > > I've 3 nodes, all services are collocated. > > > >> > >> # ceph nfs cluster create tentacle-nfs "srvceph01,srvceph02,srvceph03" > > --ingress --virtual_ip 10.200.24.89 --ingress-mode haproxy-standard > --port > > 12049 > > > > > > When I create the nfs cluster (with HA, haproxy + keepalived). 3 > > nfs-tentacle-nfs containers are created, one on each node, but they do > not > > stay alive, they start then disappear, again and again, and in the end > just > > one is created, but it does not really work, HAProxy shows that all > > backends are down. > > > > What I've got from the vanished containers is that the nfs service is > > looking for rpc-bind which does not exist because ganesha's config file > > enables both nfs v3 and v4. > > > > # ceph orch ps --service_name nfs.tentacle-nfs > >> NAME HOST PORTS STATUS > >> REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID > >> nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 unknown > >> 5m ago 6m - - <unknown> <unknown> <unknown> > >> nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 running (5m) > >> 5m ago 6m 15.0M - 5.9 c22719cb0d76 dcb1eab045cd > >> nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 running (4m) > >> 4m ago 6m 14.6M - 5.9 c22719cb0d76 b11d32f4b0cc > > > > > > > > # ceph orch ps > >> NAME HOST PORTS > >> STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE > ID > >> CONTAINER ID > >> ... > >> haproxy.nfs.tentacle-nfs.srvceph01.uvueop srvceph01 *:12049,19049 > >> running (6m) 4m ago 6m 11.5M - 2.3.17-d1c9119 > >> e85424b0d443 d4a840202ae0 > >> haproxy.nfs.tentacle-nfs.srvceph02.xujdir srvceph02 *:12049,19049 > >> running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 > >> e85424b0d443 747cf2052cd5 > >> haproxy.nfs.tentacle-nfs.srvceph03.snapkl srvceph03 *:12049,19049 > >> running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 > >> e85424b0d443 3c17cb2a4757 > >> keepalived.nfs.tentacle-nfs.srvceph01.phscfe srvceph01 > >> running (6m) 4m ago 6m 19.9M - 2.2.4 > >> 4a3a1ff181d9 d053e0b91210 > >> keepalived.nfs.tentacle-nfs.srvceph02.ubwgoo srvceph02 > >> running (6m) 5m ago 6m 20.1M - 2.2.4 > >> 4a3a1ff181d9 6ad75ccccaa0 > >> keepalived.nfs.tentacle-nfs.srvceph03.phbgis srvceph03 > >> running (6m) 5m ago 6m 20.1M - 2.2.4 > >> 4a3a1ff181d9 9efebd8d5ba5 > >> mds.cephfs.srvceph01.tyrask srvceph01 > >> running (17m) 4m ago 5d 32.4M - 20.2.2 > >> c22719cb0d76 2f89b9d86343 > >> mds.cephfs.srvceph02.ebyvdg srvceph02 > >> running (17m) 5m ago 5d 33.1M - 20.2.2 > >> c22719cb0d76 5afe19ec45b8 > >> mds.cephfs.srvceph03.miznhi srvceph03 > >> running (17m) 5m ago 5d 146M - 20.2.2 > >> c22719cb0d76 6ee9a906d887 > >> ... > >> nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 > >> unknown 5m ago 7m - - <unknown> > <unknown> > >> <unknown> > >> nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 > >> running (5m) 5m ago 7m 15.0M - 5.9 > >> c22719cb0d76 dcb1eab045cd > >> nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 > >> running (4m) 4m ago 7m 14.6M - 5.9 > >> c22719cb0d76 b11d32f4b0cc > > > > ... > > > > > > # podman logs nfs-tentacle-container > > > > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_start_grace :STATE :EVENT :grace reload client info completed from > > backend > > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > > count(0) > > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > rados_cluster_grace_enforcing :CLIENT ID :EVENT > > :rados_cluster_grace_enforcing: ret=-45 > > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_start_grace :STATE :EVENT :grace reload client info completed from > > backend > > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > > count(0) > > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > rados_cluster_grace_enforcing :CLIENT ID :EVENT > > :rados_cluster_grace_enforcing: ret=-45 > > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_start_grace :STATE :EVENT :grace reload client info completed from > > backend > > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > > count(0) > > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > > rados_cluster_grace_enforcing :CLIENT ID :EVENT > > :rados_cluster_grace_enforcing: ret=-45 > > > > > > > > After > > > >> # ceph orch restart nfs.tentacle-nfs > > > > > > I got this > > > >> # podman logs nfs-tentacle-container > > > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > init_logging :LOG :NULL :LOG: Setting log level for all components to > > NIV_EVENT > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.9 > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file > > successfully parsed > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > monitoring_init :NFS STARTUP :EVENT :Init monitoring at 0.0.0.0:9587 > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit > > on FDs to 1048576. > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > > 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > nfs_start_grace :STATE :EVENT :grace reload client info completed from > > backend > > 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > > count(0) > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > rados_cluster_grace_enforcing :CLIENT ID :EVENT > > :rados_cluster_grace_enforcing: ret=0 > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > main :NFS STARTUP :WARN :No export entries found in configuration file > !!! > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > config_errs_to_log :CONFIG :WARN :Config File > > (/etc/ganesha/ganesha.conf:9): Unknown parameter > (allow_set_io_flusher_fail) > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > config_errs_to_log :CONFIG :WARN :Config File > > (/etc/ganesha/ganesha.conf:28): Unknown block (RADOS_URLS) > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > config_errs_to_log :CONFIG :WARN :Config File > > (/etc/ganesha/ganesha.conf:33): Unknown block (RGW) > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully > > removed for proper quota management in FSAL > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: > > > cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to > > socket /run/dbus/system_bus_socket: No such file or directory) > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path > > called with no DBUS connection > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path > > called with no DBUS connection > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path > > called with no DBUS connection > > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs > > *04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > > __Register_program :DISP :MAJ :Cannot register NFS V3 on TCP > > <--------------------------* > > > > > > > > podman exec -it f2e2c668e0e3 bash > >> [root@srvceph01 /]# cat etc/ganesha/ganesha.conf > >> # This file is generated by cephadm. > >> NFS_CORE_PARAM { > >> Enable_NLM = false; > >> Enable_RQUOTA = false; > >> * Protocols = 3, 4; > >> <---------------------------------------------------* > >> mount_path_pseudo = true; > >> Enable_UDP = false; > >> NFS_Port = 22049; > >> allow_set_io_flusher_fail = true; > >> Monitoring_Port = 9587; > >> } > >> NFSv4 { > >> Delegations = false; > >> RecoveryBackend = "rados_cluster"; > >> Minor_Versions = 1, 2; > >> Server_Scope = > "bcf27a48-8a8d-11f1-bdd3-fc607038078a-tentacle-nfs"; > >> IdmapConf = "/etc/ganesha/idmap.conf"; > >> } > >> RADOS_KV { > >> UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; > >> nodeid = 2; > >> pool = ".nfs"; > >> namespace = "tentacle-nfs"; > >> } > >> RADOS_URLS { > >> UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; > >> watch_url = "rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs"; > >> } > >> RGW { > >> cluster = "ceph"; > >> name = "client.nfs.tentacle-nfs.2.0.srvceph01.ahcayv-rgw"; > >> } > >> %url rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs > > > > > > > > I am not sure if this is the cause of the problem, so I thought, why not > > enable nfsv3 even if I'm not going to use it? but... > > Ceph's documentation mentions (tentacle version selected) : > > https://docs.ceph.com/en/tentacle/mgr/nfs/#create-nfs-ganesha-cluster > > Create NFS Ganesha Cluster > > > > ceph nfs cluster create <cluster_id> .... [--enable-nfsv3] > > > > The --enable-nfsv3 is not recognized, I've tried to use it to enable > > nfsv3 (as a workaround), but it is not recognized. > > > > I've also tried to use this from > > https://docs.ceph.com/en/tentacle/cephadm/services/nfs/#cephadm-ha-nfs > > : > > > > Service Specification > > > > service_type: nfsservice_id: mynfsplacement: hosts: - host1 - > > host2spec: port: 12345 monitoring_port: 567 enable_nfsv3: true > > <------------------- not recognized > > > > The enable_nfsv3 is not recognized. > > > > If this is the cause of the problem, then I'm stuck, I don't know how to > > disable nfsv3 on ganesha nor how to enable nfsv3 with cephadm !!!!! > > > > > > Regards. > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: [Ceph tentacle 20.2.2] problem with nfs ganesha and nfsv3
by Eugen Block 04 Aug '26

04 Aug '26

Hi, look at this thread [0] and the corresponding links provided in there [1], [2]. Regards, Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/LG2EGDZMIW… [1] https://tracker.ceph.com/issues/74307 [2] https://tracker.ceph.com/issues/74191 Zitat von wodel youchi via ceph-users <ceph-users(a)ceph.io>: > Hi, > > I'm using tentacle 20.2.2 with podman on Rocky Linux 9. > I'm trying to configure nfs-ganesha, but the nfs-tentacle containers won't > stay up. > I've 3 nodes, all services are collocated. > >> >> # ceph nfs cluster create tentacle-nfs "srvceph01,srvceph02,srvceph03" > --ingress --virtual_ip 10.200.24.89 --ingress-mode haproxy-standard --port > 12049 > > > When I create the nfs cluster (with HA, haproxy + keepalived). 3 > nfs-tentacle-nfs containers are created, one on each node, but they do not > stay alive, they start then disappear, again and again, and in the end just > one is created, but it does not really work, HAProxy shows that all > backends are down. > > What I've got from the vanished containers is that the nfs service is > looking for rpc-bind which does not exist because ganesha's config file > enables both nfs v3 and v4. > > # ceph orch ps --service_name nfs.tentacle-nfs >> NAME HOST PORTS STATUS >> REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID >> nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 unknown >> 5m ago 6m - - <unknown> <unknown> <unknown> >> nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 running (5m) >> 5m ago 6m 15.0M - 5.9 c22719cb0d76 dcb1eab045cd >> nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 running (4m) >> 4m ago 6m 14.6M - 5.9 c22719cb0d76 b11d32f4b0cc > > > > # ceph orch ps >> NAME HOST PORTS >> STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID >> CONTAINER ID >> ... >> haproxy.nfs.tentacle-nfs.srvceph01.uvueop srvceph01 *:12049,19049 >> running (6m) 4m ago 6m 11.5M - 2.3.17-d1c9119 >> e85424b0d443 d4a840202ae0 >> haproxy.nfs.tentacle-nfs.srvceph02.xujdir srvceph02 *:12049,19049 >> running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 >> e85424b0d443 747cf2052cd5 >> haproxy.nfs.tentacle-nfs.srvceph03.snapkl srvceph03 *:12049,19049 >> running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 >> e85424b0d443 3c17cb2a4757 >> keepalived.nfs.tentacle-nfs.srvceph01.phscfe srvceph01 >> running (6m) 4m ago 6m 19.9M - 2.2.4 >> 4a3a1ff181d9 d053e0b91210 >> keepalived.nfs.tentacle-nfs.srvceph02.ubwgoo srvceph02 >> running (6m) 5m ago 6m 20.1M - 2.2.4 >> 4a3a1ff181d9 6ad75ccccaa0 >> keepalived.nfs.tentacle-nfs.srvceph03.phbgis srvceph03 >> running (6m) 5m ago 6m 20.1M - 2.2.4 >> 4a3a1ff181d9 9efebd8d5ba5 >> mds.cephfs.srvceph01.tyrask srvceph01 >> running (17m) 4m ago 5d 32.4M - 20.2.2 >> c22719cb0d76 2f89b9d86343 >> mds.cephfs.srvceph02.ebyvdg srvceph02 >> running (17m) 5m ago 5d 33.1M - 20.2.2 >> c22719cb0d76 5afe19ec45b8 >> mds.cephfs.srvceph03.miznhi srvceph03 >> running (17m) 5m ago 5d 146M - 20.2.2 >> c22719cb0d76 6ee9a906d887 >> ... >> nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 >> unknown 5m ago 7m - - <unknown> <unknown> >> <unknown> >> nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 >> running (5m) 5m ago 7m 15.0M - 5.9 >> c22719cb0d76 dcb1eab045cd >> nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 >> running (4m) 4m ago 7m 14.6M - 5.9 >> c22719cb0d76 b11d32f4b0cc > > ... > > > # podman logs nfs-tentacle-container > > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :grace reload client info completed from > backend > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > count(0) > 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > rados_cluster_grace_enforcing :CLIENT ID :EVENT > :rados_cluster_grace_enforcing: ret=-45 > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :grace reload client info completed from > backend > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > count(0) > 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > rados_cluster_grace_enforcing :CLIENT ID :EVENT > :rados_cluster_grace_enforcing: ret=-45 > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :grace reload client info completed from > backend > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > count(0) > 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] > rados_cluster_grace_enforcing :CLIENT ID :EVENT > :rados_cluster_grace_enforcing: ret=-45 > > > > After > >> # ceph orch restart nfs.tentacle-nfs > > > I got this > >> # podman logs nfs-tentacle-container > > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > init_logging :LOG :NULL :LOG: Setting log level for all components to > NIV_EVENT > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.9 > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file > successfully parsed > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > monitoring_init :NFS STARTUP :EVENT :Init monitoring at 0.0.0.0:9587 > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit > on FDs to 1048576. > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. > 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 > 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > nfs_start_grace :STATE :EVENT :grace reload client info completed from > backend > 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid > count(0) > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > rados_cluster_grace_enforcing :CLIENT ID :EVENT > :rados_cluster_grace_enforcing: ret=0 > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > main :NFS STARTUP :WARN :No export entries found in configuration file !!! > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > config_errs_to_log :CONFIG :WARN :Config File > (/etc/ganesha/ganesha.conf:9): Unknown parameter (allow_set_io_flusher_fail) > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > config_errs_to_log :CONFIG :WARN :Config File > (/etc/ganesha/ganesha.conf:28): Unknown block (RADOS_URLS) > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > config_errs_to_log :CONFIG :WARN :Config File > (/etc/ganesha/ganesha.conf:33): Unknown block (RGW) > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully > removed for proper quota management in FSAL > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: > cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to > socket /run/dbus/system_bus_socket: No such file or directory) > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path > called with no DBUS connection > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path > called with no DBUS connection > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path > called with no DBUS connection > 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs > *04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] > __Register_program :DISP :MAJ :Cannot register NFS V3 on TCP > <--------------------------* > > > > podman exec -it f2e2c668e0e3 bash >> [root@srvceph01 /]# cat etc/ganesha/ganesha.conf >> # This file is generated by cephadm. >> NFS_CORE_PARAM { >> Enable_NLM = false; >> Enable_RQUOTA = false; >> * Protocols = 3, 4; >> <---------------------------------------------------* >> mount_path_pseudo = true; >> Enable_UDP = false; >> NFS_Port = 22049; >> allow_set_io_flusher_fail = true; >> Monitoring_Port = 9587; >> } >> NFSv4 { >> Delegations = false; >> RecoveryBackend = "rados_cluster"; >> Minor_Versions = 1, 2; >> Server_Scope = "bcf27a48-8a8d-11f1-bdd3-fc607038078a-tentacle-nfs"; >> IdmapConf = "/etc/ganesha/idmap.conf"; >> } >> RADOS_KV { >> UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; >> nodeid = 2; >> pool = ".nfs"; >> namespace = "tentacle-nfs"; >> } >> RADOS_URLS { >> UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; >> watch_url = "rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs"; >> } >> RGW { >> cluster = "ceph"; >> name = "client.nfs.tentacle-nfs.2.0.srvceph01.ahcayv-rgw"; >> } >> %url rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs > > > > I am not sure if this is the cause of the problem, so I thought, why not > enable nfsv3 even if I'm not going to use it? but... > Ceph's documentation mentions (tentacle version selected) : > https://docs.ceph.com/en/tentacle/mgr/nfs/#create-nfs-ganesha-cluster > Create NFS Ganesha Cluster > > ceph nfs cluster create <cluster_id> .... [--enable-nfsv3] > > The --enable-nfsv3 is not recognized, I've tried to use it to enable > nfsv3 (as a workaround), but it is not recognized. > > I've also tried to use this from > https://docs.ceph.com/en/tentacle/cephadm/services/nfs/#cephadm-ha-nfs > : > > Service Specification > > service_type: nfsservice_id: mynfsplacement: hosts: - host1 - > host2spec: port: 12345 monitoring_port: 567 enable_nfsv3: true > <------------------- not recognized > > The enable_nfsv3 is not recognized. > > If this is the cause of the problem, then I'm stuck, I don't know how to > disable nfsv3 on ganesha nor how to enable nfsv3 with cephadm !!!!! > > > Regards. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

CERTIFICATE_VERIFY_FAILED ?
by listy 04 Aug '26

04 Aug '26

Hi guys I'm thinking I've my alerts configured a ok yet clusters warns: ... CEPH status is: HEALTH_WARN unable to send alert email [WRN] ALERTS_SMTP_ERROR: unable to send alert email [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:998) ... Do you know what I might be missing or getting something wrong? many thanks, L.

1 0

[Ceph tentacle 20.2.2] problem with nfs ganesha and nfsv3
by wodel youchi 04 Aug '26

04 Aug '26

Hi, I'm using tentacle 20.2.2 with podman on Rocky Linux 9. I'm trying to configure nfs-ganesha, but the nfs-tentacle containers won't stay up. I've 3 nodes, all services are collocated. > > # ceph nfs cluster create tentacle-nfs "srvceph01,srvceph02,srvceph03" --ingress --virtual_ip 10.200.24.89 --ingress-mode haproxy-standard --port 12049 When I create the nfs cluster (with HA, haproxy + keepalived). 3 nfs-tentacle-nfs containers are created, one on each node, but they do not stay alive, they start then disappear, again and again, and in the end just one is created, but it does not really work, HAProxy shows that all backends are down. What I've got from the vanished containers is that the nfs service is looking for rpc-bind which does not exist because ganesha's config file enables both nfs v3 and v4. # ceph orch ps --service_name nfs.tentacle-nfs > NAME HOST PORTS STATUS > REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID > nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 unknown > 5m ago 6m - - <unknown> <unknown> <unknown> > nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 running (5m) > 5m ago 6m 15.0M - 5.9 c22719cb0d76 dcb1eab045cd > nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 running (4m) > 4m ago 6m 14.6M - 5.9 c22719cb0d76 b11d32f4b0cc # ceph orch ps > NAME HOST PORTS > STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID > CONTAINER ID > ... > haproxy.nfs.tentacle-nfs.srvceph01.uvueop srvceph01 *:12049,19049 > running (6m) 4m ago 6m 11.5M - 2.3.17-d1c9119 > e85424b0d443 d4a840202ae0 > haproxy.nfs.tentacle-nfs.srvceph02.xujdir srvceph02 *:12049,19049 > running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 > e85424b0d443 747cf2052cd5 > haproxy.nfs.tentacle-nfs.srvceph03.snapkl srvceph03 *:12049,19049 > running (7m) 5m ago 7m 11.5M - 2.3.17-d1c9119 > e85424b0d443 3c17cb2a4757 > keepalived.nfs.tentacle-nfs.srvceph01.phscfe srvceph01 > running (6m) 4m ago 6m 19.9M - 2.2.4 > 4a3a1ff181d9 d053e0b91210 > keepalived.nfs.tentacle-nfs.srvceph02.ubwgoo srvceph02 > running (6m) 5m ago 6m 20.1M - 2.2.4 > 4a3a1ff181d9 6ad75ccccaa0 > keepalived.nfs.tentacle-nfs.srvceph03.phbgis srvceph03 > running (6m) 5m ago 6m 20.1M - 2.2.4 > 4a3a1ff181d9 9efebd8d5ba5 > mds.cephfs.srvceph01.tyrask srvceph01 > running (17m) 4m ago 5d 32.4M - 20.2.2 > c22719cb0d76 2f89b9d86343 > mds.cephfs.srvceph02.ebyvdg srvceph02 > running (17m) 5m ago 5d 33.1M - 20.2.2 > c22719cb0d76 5afe19ec45b8 > mds.cephfs.srvceph03.miznhi srvceph03 > running (17m) 5m ago 5d 146M - 20.2.2 > c22719cb0d76 6ee9a906d887 > ... > nfs.tentacle-nfs.0.0.srvceph03.ayezlm srvceph03 *:22049 > unknown 5m ago 7m - - <unknown> <unknown> > <unknown> > nfs.tentacle-nfs.1.0.srvceph02.kbmdos srvceph02 *:22049 > running (5m) 5m ago 7m 15.0M - 5.9 > c22719cb0d76 dcb1eab045cd > nfs.tentacle-nfs.2.0.srvceph01.ahcayv srvceph01 *:22049 > running (4m) 4m ago 7m 14.6M - 5.9 > c22719cb0d76 b11d32f4b0cc ... # podman logs nfs-tentacle-container 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 04/08/2026 08:39:44 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 04/08/2026 08:39:48 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 04/08/2026 08:39:52 : epoch 6a71a43c : srvceph01 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=-45 After > # ceph orch restart nfs.tentacle-nfs I got this > # podman logs nfs-tentacle-container 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] init_logging :LOG :NULL :LOG: Setting log level for all components to NIV_EVENT 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.9 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] monitoring_init :NFS STARTUP :EVENT :Init monitoring at 0.0.0.0:9587 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit on FDs to 1048576. 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. 04/08/2026 08:40:46 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 04/08/2026 08:40:52 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=0 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!! 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:9): Unknown parameter (allow_set_io_flusher_fail) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:28): Unknown block (RADOS_URLS) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:33): Unknown block (RGW) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] lower_my_caps :NFS STARTUP :EVENT :currently set capabilities are: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap=ep 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory) 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection 04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs *04/08/2026 08:40:55 : epoch 6a71a58e : srvceph01 : ganesha.nfsd-2[main] __Register_program :DISP :MAJ :Cannot register NFS V3 on TCP <--------------------------* podman exec -it f2e2c668e0e3 bash > [root@srvceph01 /]# cat etc/ganesha/ganesha.conf > # This file is generated by cephadm. > NFS_CORE_PARAM { > Enable_NLM = false; > Enable_RQUOTA = false; > * Protocols = 3, 4; > <---------------------------------------------------* > mount_path_pseudo = true; > Enable_UDP = false; > NFS_Port = 22049; > allow_set_io_flusher_fail = true; > Monitoring_Port = 9587; > } > NFSv4 { > Delegations = false; > RecoveryBackend = "rados_cluster"; > Minor_Versions = 1, 2; > Server_Scope = "bcf27a48-8a8d-11f1-bdd3-fc607038078a-tentacle-nfs"; > IdmapConf = "/etc/ganesha/idmap.conf"; > } > RADOS_KV { > UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; > nodeid = 2; > pool = ".nfs"; > namespace = "tentacle-nfs"; > } > RADOS_URLS { > UserId = "nfs.tentacle-nfs.2.0.srvceph01.ahcayv"; > watch_url = "rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs"; > } > RGW { > cluster = "ceph"; > name = "client.nfs.tentacle-nfs.2.0.srvceph01.ahcayv-rgw"; > } > %url rados://.nfs/tentacle-nfs/conf-nfs.tentacle-nfs I am not sure if this is the cause of the problem, so I thought, why not enable nfsv3 even if I'm not going to use it? but... Ceph's documentation mentions (tentacle version selected) : https://docs.ceph.com/en/tentacle/mgr/nfs/#create-nfs-ganesha-cluster Create NFS Ganesha Cluster ceph nfs cluster create <cluster_id> .... [--enable-nfsv3] The --enable-nfsv3 is not recognized, I've tried to use it to enable nfsv3 (as a workaround), but it is not recognized. I've also tried to use this from https://docs.ceph.com/en/tentacle/cephadm/services/nfs/#cephadm-ha-nfs : Service Specification service_type: nfsservice_id: mynfsplacement: hosts: - host1 - host2spec: port: 12345 monitoring_port: 567 enable_nfsv3: true <------------------- not recognized The enable_nfsv3 is not recognized. If this is the cause of the problem, then I'm stuck, I don't know how to disable nfsv3 on ganesha nor how to enable nfsv3 with cephadm !!!!! Regards.

1 0

Re: Using CephFS for encrypted Kubernetes volumes
by Eugen Block 04 Aug '26

04 Aug '26

Hi, I haven't tried it myself yet so I don't really have an answer. But I noticed that your OSD caps may be incomplete wrt x permissions: > osd_caps: 'allow rwx namespace=k8s-test tag cephfs metadata=root, > allow rw namespace=k8s-test tag cephfs data=root' The second allow statement doesn't have an "x". Maybe that's the issue why the workaround fails? Regards, Eugen Zitat von Stefan Möding via ceph-users <ceph-users(a)ceph.io>: > Hi, > > we are trying to use Ceph (Tentacle 20.2.2) as storage provider for > multiple Kubernetes clusters. We are trying to implement the following > security measures: > > * Each Kubernetes cluster should use a dedicated RADOS namespace for > its objects. > * The volumes should use fscrypt encryption. > > We can deploy pods successfully as long as we don’t try to enable > fscrypt. With encryption enabled the pod shows the following logs: > > AttachVolume.Attach succeeded for volume > "pvc-0923605d-3993-4d99-a4a3-d41749ead831" > MountVolume.MountDevice failed for volume > "pvc-0923605d-3993-4d99-a4a3-d41749ead831" : rpc error: code = > Internal desc = failed to lock volume ID > 64ab25ab-9d66-4508-be74-13c02825f40e: rados: ret=-1, Operation not > permitted > > There is a closed issue that mentions a length limitation when creating > a lock to initialize the encryption: > https://github.com/ceph/ceph-csi/issues/5419 > > I also tried the mentioned workaround but that does not help. > > There has also been a discussion that the ’x’ permission is needed in > the OSD capabilities, which we have set: > > mon_caps: 'allow r fsname=root' > mgr_caps: 'allow rw' > osd_caps: 'allow rwx namespace=k8s-test tag cephfs metadata=root, > allow rw namespace=k8s-test tag cephfs data=root' > mds_caps: 'allow r fsname=root path=/volumes, allow rws fsname=root > path=/volumes/k8s-test' > > Does anybody have a working setup with encryption and RADOS namespaces? > > -- > Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph Steering Committee Meeting 2026-08-03
by Ilya Dryomov 03 Aug '26

03 Aug '26

Hello, - first public foundation board meeting recording is available (Neha) - https://www.youtube.com/watch?v=gPyu9KLVD64 - CDS is starting this week! (Anthony) - https://ceph.io/en/community/events/2026/ceph-developer-summit-vampire/ - performance session led by Mark Nelson is yet to be scheduled, the post will be updated when that happens - upcoming Ceph Days PSA (Patrick) - Zurich: https://ceph.io/en/community/events/2026/ceph-days-zurich/ - Chicago: https://ceph.io/en/community/events/2026/ceph-days-chicago/ - merge window process (Ilya, Patrick) - the date in the calendar is the "branch closes on" not the "PR must be ready for QA by" date - this is in large part due to the fact that the time that it takes for the PR to be picked up into a QA run and the time that it takes for the test results to be reviewed are currently outside of the release manager's control - Patrick will be introducing a new needs-qa alike automated label for backport PRs to formalize the "PR is ready for QA" requirements and make such PRs more discoverable - component leads are free to do their own QA runs when needed - 21.1.1 status - merge window closes in two days - barring negative user feedback or last minute blockers this would be the last umbrella RC - "happy path" tentative release date remains to be Aug 31 - squid EOL plan - at least one merge window/release after umbrella is out Thanks, Ilya

1 0

Re: How to administer Ceph from any node without bootstrap dependency
by Wannes Smet 03 Aug '26

03 Aug '26

Hi, Also make sure every node has an _admin label. (ceph orch host label add <hostname> _admin. Verify with ceph orch host ls. Not proud of it, but I once chown-ed /var/lib/ceph on mulptiple nodes. Due to that, our Ceph cluster lost quorum because all the containers crashed. A learning experience I'll never forget #facepalm. I'll definitely think twice before I manupulate /var/lib/ceph ever again ;) . Only if your cluster is 100% test with no production workload on it: Why not try to shut down one node and see what happens on the surviving nodes. Do you get back to HEALTH_OK after you restart the downed node? It should and if it doesn't you're lucky to find out before you went in production. Also, do a "rolling" reboot of all of your nodes. Do you get back to HEALTH_OK after a reboot has finished? This for all the nodes? Anything else breaks if you shut down one node? Wannes. ________________________________ From: Eugen Block via ceph-users <ceph-users(a)ceph.io> Sent: Tuesday, July 21, 2026 22:31 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: How to administer Ceph from any node without bootstrap dependency Hi, Ceph is highly-available by design. So if you have three MON nodes (and two MGR services deployed along them) you're good as long as you have quorum and an admin keyring on more than one node. With cephadm you can manage the presence of ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, Ceph is highly-available by design. So if you have three MON nodes (and two MGR services deployed along them) you're good as long as you have quorum and an admin keyring on more than one node. With cephadm you can manage the presence of the admin keyring via orchestrator and the _admin label [0]. That enables you to manage the cluster from any admin node. Not sure what you mean by "bootstrap file" though... If you'd synced the /var/lib/ceph directory, you'd overwrite all the daemons but one, including keyring etc. So please don't do that, that would break your cluster. Regards, Eugen [0] https://urldefense.com/v3/__https://docs.ceph.com/en/latest/cephadm/operati… Zitat von Michel Le Cocq via ceph-users <ceph-users(a)ceph.io>: > Hi all, > > I am currently deploying a Ceph cluster with cephadm on 3 nodes with > MON, MGR, and MDS roles. OSDs will be added later. > > My goal is for these 3 nodes to be fully interchangeable, with the > ability to administer the cluster from any of them without risk if a > node fails. > > I have identified that the solution involves synchronizing > /var/lib/ceph/ (excluding the bootstrap file) across all nodes. > > However: > > - Is this the right approach to achieve this goal? > > - Which tool should I use for reliable bidirectional synchronization? > > - What are the potential risks (conflicts, corruption) and precautions > to take? > > Have you implemented such a configuration in production before? > > Thank you for your advice. > > -- > Michel > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0