Hej,

i've been struggling with CEPH and kerberized NFS4 and HA for a while but now seem to have a working solution with CEPH 18.2.4 ... right now in stress testing phase.

Manually set up ganesha servers - no cephadm, no containers. HA is done with keepalived.

The current solution works with only one ganesha server running at a time. keepalived switches ip adresses and starts a new ganesha server on another host in case of host or ganesha service failure. Failover and switchback works within seconds.

Setup was a bit cumbersome and trial and error: compiling matchings versions of CEPH and Ganesha (6.5), scripts for keepalived state logic to monitor and start/stop ganesha services.

Probably could be done much more elegant ... work in progress ...

Cheers, toBias

From: "Devin A. Bougie" <devin.bougie@cornell.edu>
To: "ceph-users" <ceph-users@ceph.io>
Sent: Wednesday, 23 April, 2025 00:19:01
Subject: [ceph-users] Help with HA NFS

Hello,

We’ve found that if we lose one of the nfs.cephfs service daemons in our cephadm 19.2.2 cluster, all NFS traffic is blocked until either:
- the down nfs.cephfs daemon is restarted
- or we reconfigure the placement of the nfs.cephs service to not use the affected host.  After this, the ingress.nfs.cephfs service is automatically reconfigured and everything resumes

Our current setup follows the "HIGH-AVAILABILITY NFS” documentation, which gives us an ingress.nfs.cephfs service with the haproxy and keepalived daemons and an nfs.cephfs service for the actual nfs daemons.  This service was deployed using:
ceph nfs cluster create cephfs "label:_admin" --ingress --virtual_ip virtual_ip

And then we updated the ingress.nfs.cephfs service to only deploy a single service (which in this case, results in two daemons on a single host).

This gives us the following:
———
[root@cephman1 ~]# ceph orch ls --service_name=ingress.nfs.cephfs --export
service_type: ingress
service_id: nfs.cephfs
service_name: ingress.nfs.cephfs
placement:
  count: 1
  label: _admin
spec:
  backend_service: nfs.cephfs
  first_virtual_router_id: 50
  frontend_port: 2049
  monitor_port: 9049
  virtual_ip: 128.84.45.48/22

[root@cephman1 ~]# ceph orch ls --service_name=nfs.cephfs --export
service_type: nfs
service_id: cephfs
service_name: nfs.cephfs
placement:
  label: _admin
spec:
  port: 12049
———

Can anyone show us the config for a true “HA” nfs service where they can lose any single host without impacting access to the NFS export from clients?  I would expect to be able to lose the host running the ingress.nfs.cephfs service, and have it automatically restarted on a different host.  Likewise, I would expect to be able to lose an nfs.cephs daemon without impacting access to the export.

Or should we be taking a completely different approach and move our NFS service out of Ceph and into our pacemaker / corosync cluster?

Sorry if this sounds redundant to questions I’ve previously asked, but we’ve reconfigured things a little and it feels like we’re getting closer with each attempt?

Many thanks,
Devin