I just tried it with 3 keepalive daemons and one nfs daemon, it doesn't really work because all three hosts have the virtual IP assigned, preventing my client from mounting. So this doesn't really work as a workaround, it seems.
That's a bit surprising. The keepalive daemons are meant to communicate to each other and have only one maintain the VIP while the others are in a standby state. Do you see anything in the logs about them communicating to each other? Also, in the keepalive conf for each keepalive, do you see one set with a higher "priority" setting than the other? On Tue, Mar 25, 2025 at 11:18 AM Eugen Block <eblock@nde.ag> wrote:
Thanks, Adam. I just tried it with 3 keepalive daemons and one nfs daemon, it doesn't really work because all three hosts have the virtual IP assigned, preventing my client from mounting. So this doesn't really work as a workaround, it seems. I feel like the proper solution would be to include keepalive in the list of RESCHEDULE_FROM_OFFLINE_HOSTS_TYPES.
Zitat von Adam King <adking@redhat.com>:
Which daemons get moved around like that is controlled by
which appears to only include nfs and haproxy, so maybe this keepalive only case was missed in that sense. I do think that you could alter the placement of the ingress service to just match all the hosts and it ought to work though. The only reason we require specifying a set of hosts with a count field lower than the number of hosts matching the placement for nfs is that ganesha has no transparent state migration. Keepalive on the other hand should work fine with "extra" daemons deployed in order to be highly available.
On Tue, Mar 25, 2025 at 10:06 AM Malte Stroem <malte.stroem@gmail.com> wrote:
Hi Eugen,
yes, for me it's kind of "test-setting" for small setups.
Doc says:
Setting --ingress-mode keepalive-only deploys a simplified ingress service that provides a virtual IP with the nfs server directly binding to that virtual IP and leaves out any sort of load balancing or traffic redirection. This setup will restrict users to deploying only 1 nfs daemon as multiple cannot bind to the same port on the virtual IP.
Best, Malte
On 25.03.25 13:46, Eugen Block wrote:
Yeah, it seems to work without the "keepalive-only" flag, at least from a first test. So keepalive-only is not working properly, it seems? Should I create a tracker for that or am I misunderstanding its
https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/utils.py#L30 , purpose?
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hi Eugen,
try omitting
--ingress-mode keepalive-only
like this
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip "192.168.168.114/24"
Best, Malte
On 25.03.25 13:25, Eugen Block wrote:
Thanks for your quick response. The specs I pasted are actually the result of deploying a nfs cluster like this:
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip 192.168.168.114 --ingress-mode keepalive-only
I can try redeploying it via dashboard, but I don't have a lot of confidence that it will work differently with a failover.
Zitat von Malte Stroem <malte.stroem@gmail.com>:
> Hi Eugen, > > try deploying the NFS service like this: > > https://docs.ceph.com/en/latest/mgr/nfs/ > > Some had only success deploying it via the dashboard. > > Best, > Malte > > On 25.03.25 13:02, Eugen Block wrote: >> Hi, >> >> I'm re-evaluating NFS again, testing on a virtual cluster with >> 18.2.4. For now, I don't need haproxy so I use "keepalive_only: >> true" as described in the docs [0]. I first create the ingress >> service, wait for it to start, then create the nfs cluster. I've >> added the specs at the bottom. >> >> I can mount the export with the virtual ip. Then I just shut down >> the VM where the nfs service was running, the orchestrator >> successfully starts a nfs daemon elsewhere, but the keepalive >> daemon is not failed over. So mounting or accessing the export is >> impossible, of course. And after I power up the offline host
again,
>> nothing is "repaired", keepalive and nfs run on different servers >> until I intervene manually. This doesn't seem to work as expected, >> is this a known issue (couldn't find anything on tracker)? I have >> my doubts, but maybe it works better with haproxy? Or am I missing >> something in my configuration? >> I haven't tried with a newer release yet. I'd appreciate any comments. >> >> Thanks, >> Eugen >> >> ---snip--- >> service_type: ingress >> service_id: nfs.ebl-nfs-cephfs >> service_name: ingress.nfs.ebl-nfs-cephfs >> placement: >> count: 1 >> hosts: >> - ceph01 >> - ceph02 >> - ceph03 >> spec: >> backend_service: nfs.ebl-nfs-cephfs >> first_virtual_router_id: 50 >> keepalive_only: true >> monitor_port: 9049 >> virtual_ip: 192.168.168.114/24 >> >> >> service_type: nfs >> service_id: ebl-nfs-cephfs >> service_name: nfs.ebl-nfs-cephfs >> placement: >> count: 1 >> hosts: >> - ceph01 >> - ceph02 >> - ceph03 >> spec: >> port: 2049 >> virtual_ip: 192.168.168.114 >> ---snip--- >> >> [0] https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with- >> virtual-ip-but-no-haproxy >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io