Reef: highly-available NFS with keepalive_only
Hi, I'm re-evaluating NFS again, testing on a virtual cluster with 18.2.4. For now, I don't need haproxy so I use "keepalive_only: true" as described in the docs [0]. I first create the ingress service, wait for it to start, then create the nfs cluster. I've added the specs at the bottom. I can mount the export with the virtual ip. Then I just shut down the VM where the nfs service was running, the orchestrator successfully starts a nfs daemon elsewhere, but the keepalive daemon is not failed over. So mounting or accessing the export is impossible, of course. And after I power up the offline host again, nothing is "repaired", keepalive and nfs run on different servers until I intervene manually. This doesn't seem to work as expected, is this a known issue (couldn't find anything on tracker)? I have my doubts, but maybe it works better with haproxy? Or am I missing something in my configuration? I haven't tried with a newer release yet. I'd appreciate any comments. Thanks, Eugen ---snip--- service_type: ingress service_id: nfs.ebl-nfs-cephfs service_name: ingress.nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: backend_service: nfs.ebl-nfs-cephfs first_virtual_router_id: 50 keepalive_only: true monitor_port: 9049 virtual_ip: 192.168.168.114/24 service_type: nfs service_id: ebl-nfs-cephfs service_name: nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: port: 2049 virtual_ip: 192.168.168.114 ---snip--- [0] https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with-virtual-ip-but-...
Hi Eugen, try deploying the NFS service like this: https://docs.ceph.com/en/latest/mgr/nfs/ Some had only success deploying it via the dashboard. Best, Malte On 25.03.25 13:02, Eugen Block wrote:
Hi,
I'm re-evaluating NFS again, testing on a virtual cluster with 18.2.4. For now, I don't need haproxy so I use "keepalive_only: true" as described in the docs [0]. I first create the ingress service, wait for it to start, then create the nfs cluster. I've added the specs at the bottom.
I can mount the export with the virtual ip. Then I just shut down the VM where the nfs service was running, the orchestrator successfully starts a nfs daemon elsewhere, but the keepalive daemon is not failed over. So mounting or accessing the export is impossible, of course. And after I power up the offline host again, nothing is "repaired", keepalive and nfs run on different servers until I intervene manually. This doesn't seem to work as expected, is this a known issue (couldn't find anything on tracker)? I have my doubts, but maybe it works better with haproxy? Or am I missing something in my configuration? I haven't tried with a newer release yet. I'd appreciate any comments.
Thanks, Eugen
---snip--- service_type: ingress service_id: nfs.ebl-nfs-cephfs service_name: ingress.nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: backend_service: nfs.ebl-nfs-cephfs first_virtual_router_id: 50 keepalive_only: true monitor_port: 9049 virtual_ip: 192.168.168.114/24
service_type: nfs service_id: ebl-nfs-cephfs service_name: nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: port: 2049 virtual_ip: 192.168.168.114 ---snip---
[0] https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with- virtual-ip-but-no-haproxy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for your quick response. The specs I pasted are actually the result of deploying a nfs cluster like this: ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" --ingress --virtual_ip 192.168.168.114 --ingress-mode keepalive-only I can try redeploying it via dashboard, but I don't have a lot of confidence that it will work differently with a failover. Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hi Eugen,
try deploying the NFS service like this:
https://docs.ceph.com/en/latest/mgr/nfs/
Some had only success deploying it via the dashboard.
Best, Malte
On 25.03.25 13:02, Eugen Block wrote:
Hi,
I'm re-evaluating NFS again, testing on a virtual cluster with 18.2.4. For now, I don't need haproxy so I use "keepalive_only: true" as described in the docs [0]. I first create the ingress service, wait for it to start, then create the nfs cluster. I've added the specs at the bottom.
I can mount the export with the virtual ip. Then I just shut down the VM where the nfs service was running, the orchestrator successfully starts a nfs daemon elsewhere, but the keepalive daemon is not failed over. So mounting or accessing the export is impossible, of course. And after I power up the offline host again, nothing is "repaired", keepalive and nfs run on different servers until I intervene manually. This doesn't seem to work as expected, is this a known issue (couldn't find anything on tracker)? I have my doubts, but maybe it works better with haproxy? Or am I missing something in my configuration? I haven't tried with a newer release yet. I'd appreciate any comments.
Thanks, Eugen
---snip--- service_type: ingress service_id: nfs.ebl-nfs-cephfs service_name: ingress.nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: backend_service: nfs.ebl-nfs-cephfs first_virtual_router_id: 50 keepalive_only: true monitor_port: 9049 virtual_ip: 192.168.168.114/24
service_type: nfs service_id: ebl-nfs-cephfs service_name: nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: port: 2049 virtual_ip: 192.168.168.114 ---snip---
[0] https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with- virtual-ip-but-no-haproxy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, try omitting --ingress-mode keepalive-only like this ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" --ingress --virtual_ip "192.168.168.114/24" Best, Malte On 25.03.25 13:25, Eugen Block wrote:
Thanks for your quick response. The specs I pasted are actually the result of deploying a nfs cluster like this:
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip 192.168.168.114 --ingress-mode keepalive-only
I can try redeploying it via dashboard, but I don't have a lot of confidence that it will work differently with a failover.
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hi Eugen,
try deploying the NFS service like this:
https://docs.ceph.com/en/latest/mgr/nfs/
Some had only success deploying it via the dashboard.
Best, Malte
On 25.03.25 13:02, Eugen Block wrote:
Hi,
I'm re-evaluating NFS again, testing on a virtual cluster with 18.2.4. For now, I don't need haproxy so I use "keepalive_only: true" as described in the docs [0]. I first create the ingress service, wait for it to start, then create the nfs cluster. I've added the specs at the bottom.
I can mount the export with the virtual ip. Then I just shut down the VM where the nfs service was running, the orchestrator successfully starts a nfs daemon elsewhere, but the keepalive daemon is not failed over. So mounting or accessing the export is impossible, of course. And after I power up the offline host again, nothing is "repaired", keepalive and nfs run on different servers until I intervene manually. This doesn't seem to work as expected, is this a known issue (couldn't find anything on tracker)? I have my doubts, but maybe it works better with haproxy? Or am I missing something in my configuration? I haven't tried with a newer release yet. I'd appreciate any comments.
Thanks, Eugen
---snip--- service_type: ingress service_id: nfs.ebl-nfs-cephfs service_name: ingress.nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: backend_service: nfs.ebl-nfs-cephfs first_virtual_router_id: 50 keepalive_only: true monitor_port: 9049 virtual_ip: 192.168.168.114/24
service_type: nfs service_id: ebl-nfs-cephfs service_name: nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: port: 2049 virtual_ip: 192.168.168.114 ---snip---
[0] https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with- virtual-ip-but-no-haproxy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yeah, it seems to work without the "keepalive-only" flag, at least from a first test. So keepalive-only is not working properly, it seems? Should I create a tracker for that or am I misunderstanding its purpose? Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hi Eugen,
try omitting
--ingress-mode keepalive-only
like this
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" --ingress --virtual_ip "192.168.168.114/24"
Best, Malte
On 25.03.25 13:25, Eugen Block wrote:
Thanks for your quick response. The specs I pasted are actually the result of deploying a nfs cluster like this:
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip 192.168.168.114 --ingress-mode keepalive-only
I can try redeploying it via dashboard, but I don't have a lot of confidence that it will work differently with a failover.
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hi Eugen,
try deploying the NFS service like this:
https://docs.ceph.com/en/latest/mgr/nfs/
Some had only success deploying it via the dashboard.
Best, Malte
On 25.03.25 13:02, Eugen Block wrote:
Hi,
I'm re-evaluating NFS again, testing on a virtual cluster with 18.2.4. For now, I don't need haproxy so I use "keepalive_only: true" as described in the docs [0]. I first create the ingress service, wait for it to start, then create the nfs cluster. I've added the specs at the bottom.
I can mount the export with the virtual ip. Then I just shut down the VM where the nfs service was running, the orchestrator successfully starts a nfs daemon elsewhere, but the keepalive daemon is not failed over. So mounting or accessing the export is impossible, of course. And after I power up the offline host again, nothing is "repaired", keepalive and nfs run on different servers until I intervene manually. This doesn't seem to work as expected, is this a known issue (couldn't find anything on tracker)? I have my doubts, but maybe it works better with haproxy? Or am I missing something in my configuration? I haven't tried with a newer release yet. I'd appreciate any comments.
Thanks, Eugen
---snip--- service_type: ingress service_id: nfs.ebl-nfs-cephfs service_name: ingress.nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: backend_service: nfs.ebl-nfs-cephfs first_virtual_router_id: 50 keepalive_only: true monitor_port: 9049 virtual_ip: 192.168.168.114/24
service_type: nfs service_id: ebl-nfs-cephfs service_name: nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: port: 2049 virtual_ip: 192.168.168.114 ---snip---
[0] https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with- virtual-ip-but-no-haproxy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, yes, for me it's kind of "test-setting" for small setups. Doc says: Setting --ingress-mode keepalive-only deploys a simplified ingress service that provides a virtual IP with the nfs server directly binding to that virtual IP and leaves out any sort of load balancing or traffic redirection. This setup will restrict users to deploying only 1 nfs daemon as multiple cannot bind to the same port on the virtual IP. Best, Malte On 25.03.25 13:46, Eugen Block wrote:
Yeah, it seems to work without the "keepalive-only" flag, at least from a first test. So keepalive-only is not working properly, it seems? Should I create a tracker for that or am I misunderstanding its purpose?
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hi Eugen,
try omitting
--ingress-mode keepalive-only
like this
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip "192.168.168.114/24"
Best, Malte
On 25.03.25 13:25, Eugen Block wrote:
Thanks for your quick response. The specs I pasted are actually the result of deploying a nfs cluster like this:
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip 192.168.168.114 --ingress-mode keepalive-only
I can try redeploying it via dashboard, but I don't have a lot of confidence that it will work differently with a failover.
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hi Eugen,
try deploying the NFS service like this:
https://docs.ceph.com/en/latest/mgr/nfs/
Some had only success deploying it via the dashboard.
Best, Malte
On 25.03.25 13:02, Eugen Block wrote:
Hi,
I'm re-evaluating NFS again, testing on a virtual cluster with 18.2.4. For now, I don't need haproxy so I use "keepalive_only: true" as described in the docs [0]. I first create the ingress service, wait for it to start, then create the nfs cluster. I've added the specs at the bottom.
I can mount the export with the virtual ip. Then I just shut down the VM where the nfs service was running, the orchestrator successfully starts a nfs daemon elsewhere, but the keepalive daemon is not failed over. So mounting or accessing the export is impossible, of course. And after I power up the offline host again, nothing is "repaired", keepalive and nfs run on different servers until I intervene manually. This doesn't seem to work as expected, is this a known issue (couldn't find anything on tracker)? I have my doubts, but maybe it works better with haproxy? Or am I missing something in my configuration? I haven't tried with a newer release yet. I'd appreciate any comments.
Thanks, Eugen
---snip--- service_type: ingress service_id: nfs.ebl-nfs-cephfs service_name: ingress.nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: backend_service: nfs.ebl-nfs-cephfs first_virtual_router_id: 50 keepalive_only: true monitor_port: 9049 virtual_ip: 192.168.168.114/24
service_type: nfs service_id: ebl-nfs-cephfs service_name: nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: port: 2049 virtual_ip: 192.168.168.114 ---snip---
[0] https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with- virtual-ip-but-no-haproxy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Which daemons get moved around like that is controlled by https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/utils.py#L30, which appears to only include nfs and haproxy, so maybe this keepalive only case was missed in that sense. I do think that you could alter the placement of the ingress service to just match all the hosts and it ought to work though. The only reason we require specifying a set of hosts with a count field lower than the number of hosts matching the placement for nfs is that ganesha has no transparent state migration. Keepalive on the other hand should work fine with "extra" daemons deployed in order to be highly available. On Tue, Mar 25, 2025 at 10:06 AM Malte Stroem <malte.stroem@gmail.com> wrote:
Hi Eugen,
yes, for me it's kind of "test-setting" for small setups.
Doc says:
Setting --ingress-mode keepalive-only deploys a simplified ingress service that provides a virtual IP with the nfs server directly binding to that virtual IP and leaves out any sort of load balancing or traffic redirection. This setup will restrict users to deploying only 1 nfs daemon as multiple cannot bind to the same port on the virtual IP.
Best, Malte
On 25.03.25 13:46, Eugen Block wrote:
Yeah, it seems to work without the "keepalive-only" flag, at least from a first test. So keepalive-only is not working properly, it seems? Should I create a tracker for that or am I misunderstanding its purpose?
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hi Eugen,
try omitting
--ingress-mode keepalive-only
like this
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip "192.168.168.114/24"
Best, Malte
On 25.03.25 13:25, Eugen Block wrote:
Thanks for your quick response. The specs I pasted are actually the result of deploying a nfs cluster like this:
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip 192.168.168.114 --ingress-mode keepalive-only
I can try redeploying it via dashboard, but I don't have a lot of confidence that it will work differently with a failover.
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hi Eugen,
try deploying the NFS service like this:
https://docs.ceph.com/en/latest/mgr/nfs/
Some had only success deploying it via the dashboard.
Best, Malte
On 25.03.25 13:02, Eugen Block wrote:
Hi,
I'm re-evaluating NFS again, testing on a virtual cluster with 18.2.4. For now, I don't need haproxy so I use "keepalive_only: true" as described in the docs [0]. I first create the ingress service, wait for it to start, then create the nfs cluster. I've added the specs at the bottom.
I can mount the export with the virtual ip. Then I just shut down the VM where the nfs service was running, the orchestrator successfully starts a nfs daemon elsewhere, but the keepalive daemon is not failed over. So mounting or accessing the export is impossible, of course. And after I power up the offline host again, nothing is "repaired", keepalive and nfs run on different servers until I intervene manually. This doesn't seem to work as expected, is this a known issue (couldn't find anything on tracker)? I have my doubts, but maybe it works better with haproxy? Or am I missing something in my configuration? I haven't tried with a newer release yet. I'd appreciate any comments.
Thanks, Eugen
---snip--- service_type: ingress service_id: nfs.ebl-nfs-cephfs service_name: ingress.nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: backend_service: nfs.ebl-nfs-cephfs first_virtual_router_id: 50 keepalive_only: true monitor_port: 9049 virtual_ip: 192.168.168.114/24
service_type: nfs service_id: ebl-nfs-cephfs service_name: nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: port: 2049 virtual_ip: 192.168.168.114 ---snip---
[0] https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with- virtual-ip-but-no-haproxy _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks, Adam. I just tried it with 3 keepalive daemons and one nfs daemon, it doesn't really work because all three hosts have the virtual IP assigned, preventing my client from mounting. So this doesn't really work as a workaround, it seems. I feel like the proper solution would be to include keepalive in the list of RESCHEDULE_FROM_OFFLINE_HOSTS_TYPES. Zitat von Adam King <adking@redhat.com>:
Which daemons get moved around like that is controlled by https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/utils.py#L30, which appears to only include nfs and haproxy, so maybe this keepalive only case was missed in that sense. I do think that you could alter the placement of the ingress service to just match all the hosts and it ought to work though. The only reason we require specifying a set of hosts with a count field lower than the number of hosts matching the placement for nfs is that ganesha has no transparent state migration. Keepalive on the other hand should work fine with "extra" daemons deployed in order to be highly available.
On Tue, Mar 25, 2025 at 10:06 AM Malte Stroem <malte.stroem@gmail.com> wrote:
Hi Eugen,
yes, for me it's kind of "test-setting" for small setups.
Doc says:
Setting --ingress-mode keepalive-only deploys a simplified ingress service that provides a virtual IP with the nfs server directly binding to that virtual IP and leaves out any sort of load balancing or traffic redirection. This setup will restrict users to deploying only 1 nfs daemon as multiple cannot bind to the same port on the virtual IP.
Best, Malte
On 25.03.25 13:46, Eugen Block wrote:
Yeah, it seems to work without the "keepalive-only" flag, at least from a first test. So keepalive-only is not working properly, it seems? Should I create a tracker for that or am I misunderstanding its purpose?
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hi Eugen,
try omitting
--ingress-mode keepalive-only
like this
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip "192.168.168.114/24"
Best, Malte
On 25.03.25 13:25, Eugen Block wrote:
Thanks for your quick response. The specs I pasted are actually the result of deploying a nfs cluster like this:
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip 192.168.168.114 --ingress-mode keepalive-only
I can try redeploying it via dashboard, but I don't have a lot of confidence that it will work differently with a failover.
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hi Eugen,
try deploying the NFS service like this:
https://docs.ceph.com/en/latest/mgr/nfs/
Some had only success deploying it via the dashboard.
Best, Malte
On 25.03.25 13:02, Eugen Block wrote: > Hi, > > I'm re-evaluating NFS again, testing on a virtual cluster with > 18.2.4. For now, I don't need haproxy so I use "keepalive_only: > true" as described in the docs [0]. I first create the ingress > service, wait for it to start, then create the nfs cluster. I've > added the specs at the bottom. > > I can mount the export with the virtual ip. Then I just shut down > the VM where the nfs service was running, the orchestrator > successfully starts a nfs daemon elsewhere, but the keepalive > daemon is not failed over. So mounting or accessing the export is > impossible, of course. And after I power up the offline host again, > nothing is "repaired", keepalive and nfs run on different servers > until I intervene manually. This doesn't seem to work as expected, > is this a known issue (couldn't find anything on tracker)? I have > my doubts, but maybe it works better with haproxy? Or am I missing > something in my configuration? > I haven't tried with a newer release yet. I'd appreciate any comments. > > Thanks, > Eugen > > ---snip--- > service_type: ingress > service_id: nfs.ebl-nfs-cephfs > service_name: ingress.nfs.ebl-nfs-cephfs > placement: > count: 1 > hosts: > - ceph01 > - ceph02 > - ceph03 > spec: > backend_service: nfs.ebl-nfs-cephfs > first_virtual_router_id: 50 > keepalive_only: true > monitor_port: 9049 > virtual_ip: 192.168.168.114/24 > > > service_type: nfs > service_id: ebl-nfs-cephfs > service_name: nfs.ebl-nfs-cephfs > placement: > count: 1 > hosts: > - ceph01 > - ceph02 > - ceph03 > spec: > port: 2049 > virtual_ip: 192.168.168.114 > ---snip--- > > [0] https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with- > virtual-ip-but-no-haproxy > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, I’m not sure if this helps, and I would greatly appreciate any suggestions for improving our setup, but so far we’ve had good luck with our service deployed using: ceph nfs cluster create cephfs "label:_admin" --ingress --virtual_ip virtual_ip And then we manually updated the nfs.cephfs service this created to place the nfs daemons on our OSD nodes. This gives us the following: ——— service_type: ingress service_id: nfs.cephfs service_name: ingress.nfs.cephfs placement: label: _admin spec: backend_service: nfs.cephfs first_virtual_router_id: 50 frontend_port: 2049 monitor_port: 9049 virtual_ip: virtual_ip/prefix service_type: nfs service_id: cephfs service_name: nfs.cephfs placement: label: osd spec: port: 12049 ——— Given that we have 5 dedicated management / admin nodes and 5 separate OSD nodes, we then have: ——— [root@cephadmin1 ~]# ceph orch ls --service_name=ingress.nfs.cephfs NAME PORTS RUNNING REFRESHED AGE PLACEMENT ingress.nfs.cephfs virtual_ip:2049,9049 10/10 8m ago 10w label:_admin [root@cephadmin1 ~]# ceph orch ls --service_name=nfs.cephfs NAME PORTS RUNNING REFRESHED AGE PLACEMENT nfs.cephfs ?:12049 5/5 8m ago 7w label:osd ——— At least during testing, failover seemed to work properly. We’re still very new to Ceph, however, so would greatly appreciate knowing if anyone sees any problems with this setup or has suggestions for improvement. For example, we’re still unsure if it would be better to have the ingress.nfs.cephfs and nfs.cephfs services running on the same nodes, if one or both should be running on the dedicated OSD nodes, etc. Thanks! Devin
On Mar 25, 2025, at 11:18 AM, Eugen Block <eblock@nde.ag> wrote:
Thanks, Adam. I just tried it with 3 keepalive daemons and one nfs daemon, it doesn't really work because all three hosts have the virtual IP assigned, preventing my client from mounting. So this doesn't really work as a workaround, it seems. I feel like the proper solution would be to include keepalive in the list of RESCHEDULE_FROM_OFFLINE_HOSTS_TYPES.
Zitat von Adam King <adking@redhat.com>:
Which daemons get moved around like that is controlled by https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/utils.py#L30, which appears to only include nfs and haproxy, so maybe this keepalive only case was missed in that sense. I do think that you could alter the placement of the ingress service to just match all the hosts and it ought to work though. The only reason we require specifying a set of hosts with a count field lower than the number of hosts matching the placement for nfs is that ganesha has no transparent state migration. Keepalive on the other hand should work fine with "extra" daemons deployed in order to be highly available.
On Tue, Mar 25, 2025 at 10:06 AM Malte Stroem <malte.stroem@gmail.com> wrote:
Hi Eugen,
yes, for me it's kind of "test-setting" for small setups.
Doc says:
Setting --ingress-mode keepalive-only deploys a simplified ingress service that provides a virtual IP with the nfs server directly binding to that virtual IP and leaves out any sort of load balancing or traffic redirection. This setup will restrict users to deploying only 1 nfs daemon as multiple cannot bind to the same port on the virtual IP.
Best, Malte
On 25.03.25 13:46, Eugen Block wrote:
Yeah, it seems to work without the "keepalive-only" flag, at least from a first test. So keepalive-only is not working properly, it seems? Should I create a tracker for that or am I misunderstanding its purpose?
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hi Eugen,
try omitting
--ingress-mode keepalive-only
like this
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip "192.168.168.114/24"
Best, Malte
On 25.03.25 13:25, Eugen Block wrote:
Thanks for your quick response. The specs I pasted are actually the result of deploying a nfs cluster like this:
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip 192.168.168.114 --ingress-mode keepalive-only
I can try redeploying it via dashboard, but I don't have a lot of confidence that it will work differently with a failover.
Zitat von Malte Stroem <malte.stroem@gmail.com>:
> Hi Eugen, > > try deploying the NFS service like this: > > https://docs.ceph.com/en/latest/mgr/nfs/ > > Some had only success deploying it via the dashboard. > > Best, > Malte > > On 25.03.25 13:02, Eugen Block wrote: >> Hi, >> >> I'm re-evaluating NFS again, testing on a virtual cluster with >> 18.2.4. For now, I don't need haproxy so I use "keepalive_only: >> true" as described in the docs [0]. I first create the ingress >> service, wait for it to start, then create the nfs cluster. I've >> added the specs at the bottom. >> >> I can mount the export with the virtual ip. Then I just shut down >> the VM where the nfs service was running, the orchestrator >> successfully starts a nfs daemon elsewhere, but the keepalive >> daemon is not failed over. So mounting or accessing the export is >> impossible, of course. And after I power up the offline host again, >> nothing is "repaired", keepalive and nfs run on different servers >> until I intervene manually. This doesn't seem to work as expected, >> is this a known issue (couldn't find anything on tracker)? I have >> my doubts, but maybe it works better with haproxy? Or am I missing >> something in my configuration? >> I haven't tried with a newer release yet. I'd appreciate any comments. >> >> Thanks, >> Eugen >> >> ---snip--- >> service_type: ingress >> service_id: nfs.ebl-nfs-cephfs >> service_name: ingress.nfs.ebl-nfs-cephfs >> placement: >> count: 1 >> hosts: >> - ceph01 >> - ceph02 >> - ceph03 >> spec: >> backend_service: nfs.ebl-nfs-cephfs >> first_virtual_router_id: 50 >> keepalive_only: true >> monitor_port: 9049 >> virtual_ip: 192.168.168.114/24 >> >> >> service_type: nfs >> service_id: ebl-nfs-cephfs >> service_name: nfs.ebl-nfs-cephfs >> placement: >> count: 1 >> hosts: >> - ceph01 >> - ceph02 >> - ceph03 >> spec: >> port: 2049 >> virtual_ip: 192.168.168.114 >> ---snip--- >> >> [0] https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with- >> virtual-ip-but-no-haproxy >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I just tried it with 3 keepalive daemons and one nfs daemon, it doesn't really work because all three hosts have the virtual IP assigned, preventing my client from mounting. So this doesn't really work as a workaround, it seems.
That's a bit surprising. The keepalive daemons are meant to communicate to each other and have only one maintain the VIP while the others are in a standby state. Do you see anything in the logs about them communicating to each other? Also, in the keepalive conf for each keepalive, do you see one set with a higher "priority" setting than the other? On Tue, Mar 25, 2025 at 11:18 AM Eugen Block <eblock@nde.ag> wrote:
Thanks, Adam. I just tried it with 3 keepalive daemons and one nfs daemon, it doesn't really work because all three hosts have the virtual IP assigned, preventing my client from mounting. So this doesn't really work as a workaround, it seems. I feel like the proper solution would be to include keepalive in the list of RESCHEDULE_FROM_OFFLINE_HOSTS_TYPES.
Zitat von Adam King <adking@redhat.com>:
Which daemons get moved around like that is controlled by
which appears to only include nfs and haproxy, so maybe this keepalive only case was missed in that sense. I do think that you could alter the placement of the ingress service to just match all the hosts and it ought to work though. The only reason we require specifying a set of hosts with a count field lower than the number of hosts matching the placement for nfs is that ganesha has no transparent state migration. Keepalive on the other hand should work fine with "extra" daemons deployed in order to be highly available.
On Tue, Mar 25, 2025 at 10:06 AM Malte Stroem <malte.stroem@gmail.com> wrote:
Hi Eugen,
yes, for me it's kind of "test-setting" for small setups.
Doc says:
Setting --ingress-mode keepalive-only deploys a simplified ingress service that provides a virtual IP with the nfs server directly binding to that virtual IP and leaves out any sort of load balancing or traffic redirection. This setup will restrict users to deploying only 1 nfs daemon as multiple cannot bind to the same port on the virtual IP.
Best, Malte
On 25.03.25 13:46, Eugen Block wrote:
Yeah, it seems to work without the "keepalive-only" flag, at least from a first test. So keepalive-only is not working properly, it seems? Should I create a tracker for that or am I misunderstanding its
https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/utils.py#L30 , purpose?
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hi Eugen,
try omitting
--ingress-mode keepalive-only
like this
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip "192.168.168.114/24"
Best, Malte
On 25.03.25 13:25, Eugen Block wrote:
Thanks for your quick response. The specs I pasted are actually the result of deploying a nfs cluster like this:
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip 192.168.168.114 --ingress-mode keepalive-only
I can try redeploying it via dashboard, but I don't have a lot of confidence that it will work differently with a failover.
Zitat von Malte Stroem <malte.stroem@gmail.com>:
> Hi Eugen, > > try deploying the NFS service like this: > > https://docs.ceph.com/en/latest/mgr/nfs/ > > Some had only success deploying it via the dashboard. > > Best, > Malte > > On 25.03.25 13:02, Eugen Block wrote: >> Hi, >> >> I'm re-evaluating NFS again, testing on a virtual cluster with >> 18.2.4. For now, I don't need haproxy so I use "keepalive_only: >> true" as described in the docs [0]. I first create the ingress >> service, wait for it to start, then create the nfs cluster. I've >> added the specs at the bottom. >> >> I can mount the export with the virtual ip. Then I just shut down >> the VM where the nfs service was running, the orchestrator >> successfully starts a nfs daemon elsewhere, but the keepalive >> daemon is not failed over. So mounting or accessing the export is >> impossible, of course. And after I power up the offline host
again,
>> nothing is "repaired", keepalive and nfs run on different servers >> until I intervene manually. This doesn't seem to work as expected, >> is this a known issue (couldn't find anything on tracker)? I have >> my doubts, but maybe it works better with haproxy? Or am I missing >> something in my configuration? >> I haven't tried with a newer release yet. I'd appreciate any comments. >> >> Thanks, >> Eugen >> >> ---snip--- >> service_type: ingress >> service_id: nfs.ebl-nfs-cephfs >> service_name: ingress.nfs.ebl-nfs-cephfs >> placement: >> count: 1 >> hosts: >> - ceph01 >> - ceph02 >> - ceph03 >> spec: >> backend_service: nfs.ebl-nfs-cephfs >> first_virtual_router_id: 50 >> keepalive_only: true >> monitor_port: 9049 >> virtual_ip: 192.168.168.114/24 >> >> >> service_type: nfs >> service_id: ebl-nfs-cephfs >> service_name: nfs.ebl-nfs-cephfs >> placement: >> count: 1 >> hosts: >> - ceph01 >> - ceph02 >> - ceph03 >> spec: >> port: 2049 >> virtual_ip: 192.168.168.114 >> ---snip--- >> >> [0] https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with- >> virtual-ip-but-no-haproxy >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Okay, so I don't see anything in the keepalive log about communicating between each other. The config files are almost identical, no difference in priority, but in unicast_peer. ceph03 has no entry at all for unicast_peer, ceph02 has only ceph03 in there while ceph01 has both of the others in its unicast_peer entry. That's weird, isn't it? And the only nfs daemon is currently running on ceph02. Zitat von Adam King <adking@redhat.com>:
I just tried it with 3 keepalive daemons and one nfs daemon, it doesn't really work because all three hosts have the virtual IP assigned, preventing my client from mounting. So this doesn't really work as a workaround, it seems.
That's a bit surprising. The keepalive daemons are meant to communicate to each other and have only one maintain the VIP while the others are in a standby state. Do you see anything in the logs about them communicating to each other? Also, in the keepalive conf for each keepalive, do you see one set with a higher "priority" setting than the other?
On Tue, Mar 25, 2025 at 11:18 AM Eugen Block <eblock@nde.ag> wrote:
Thanks, Adam. I just tried it with 3 keepalive daemons and one nfs daemon, it doesn't really work because all three hosts have the virtual IP assigned, preventing my client from mounting. So this doesn't really work as a workaround, it seems. I feel like the proper solution would be to include keepalive in the list of RESCHEDULE_FROM_OFFLINE_HOSTS_TYPES.
Zitat von Adam King <adking@redhat.com>:
Which daemons get moved around like that is controlled by
which appears to only include nfs and haproxy, so maybe this keepalive only case was missed in that sense. I do think that you could alter the placement of the ingress service to just match all the hosts and it ought to work though. The only reason we require specifying a set of hosts with a count field lower than the number of hosts matching the placement for nfs is that ganesha has no transparent state migration. Keepalive on the other hand should work fine with "extra" daemons deployed in order to be highly available.
On Tue, Mar 25, 2025 at 10:06 AM Malte Stroem <malte.stroem@gmail.com> wrote:
Hi Eugen,
yes, for me it's kind of "test-setting" for small setups.
Doc says:
Setting --ingress-mode keepalive-only deploys a simplified ingress service that provides a virtual IP with the nfs server directly binding to that virtual IP and leaves out any sort of load balancing or traffic redirection. This setup will restrict users to deploying only 1 nfs daemon as multiple cannot bind to the same port on the virtual IP.
Best, Malte
On 25.03.25 13:46, Eugen Block wrote:
Yeah, it seems to work without the "keepalive-only" flag, at least from a first test. So keepalive-only is not working properly, it seems? Should I create a tracker for that or am I misunderstanding its
https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/utils.py#L30 , purpose?
Zitat von Malte Stroem <malte.stroem@gmail.com>:
Hi Eugen,
try omitting
--ingress-mode keepalive-only
like this
ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- ingress --virtual_ip "192.168.168.114/24"
Best, Malte
On 25.03.25 13:25, Eugen Block wrote: > Thanks for your quick response. The specs I pasted are actually the > result of deploying a nfs cluster like this: > > ceph nfs cluster create ebl-nfs-cephfs "1 ceph01 ceph02 ceph03" -- > ingress --virtual_ip 192.168.168.114 --ingress-mode keepalive-only > > I can try redeploying it via dashboard, but I don't have a lot of > confidence that it will work differently with a failover. > > Zitat von Malte Stroem <malte.stroem@gmail.com>: > >> Hi Eugen, >> >> try deploying the NFS service like this: >> >> https://docs.ceph.com/en/latest/mgr/nfs/ >> >> Some had only success deploying it via the dashboard. >> >> Best, >> Malte >> >> On 25.03.25 13:02, Eugen Block wrote: >>> Hi, >>> >>> I'm re-evaluating NFS again, testing on a virtual cluster with >>> 18.2.4. For now, I don't need haproxy so I use "keepalive_only: >>> true" as described in the docs [0]. I first create the ingress >>> service, wait for it to start, then create the nfs cluster. I've >>> added the specs at the bottom. >>> >>> I can mount the export with the virtual ip. Then I just shut down >>> the VM where the nfs service was running, the orchestrator >>> successfully starts a nfs daemon elsewhere, but the keepalive >>> daemon is not failed over. So mounting or accessing the export is >>> impossible, of course. And after I power up the offline host
again,
>>> nothing is "repaired", keepalive and nfs run on different servers >>> until I intervene manually. This doesn't seem to work as expected, >>> is this a known issue (couldn't find anything on tracker)? I have >>> my doubts, but maybe it works better with haproxy? Or am I missing >>> something in my configuration? >>> I haven't tried with a newer release yet. I'd appreciate any comments. >>> >>> Thanks, >>> Eugen >>> >>> ---snip--- >>> service_type: ingress >>> service_id: nfs.ebl-nfs-cephfs >>> service_name: ingress.nfs.ebl-nfs-cephfs >>> placement: >>> count: 1 >>> hosts: >>> - ceph01 >>> - ceph02 >>> - ceph03 >>> spec: >>> backend_service: nfs.ebl-nfs-cephfs >>> first_virtual_router_id: 50 >>> keepalive_only: true >>> monitor_port: 9049 >>> virtual_ip: 192.168.168.114/24 >>> >>> >>> service_type: nfs >>> service_id: ebl-nfs-cephfs >>> service_name: nfs.ebl-nfs-cephfs >>> placement: >>> count: 1 >>> hosts: >>> - ceph01 >>> - ceph02 >>> - ceph03 >>> spec: >>> port: 2049 >>> virtual_ip: 192.168.168.114 >>> ---snip--- >>> >>> [0] https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with- >>> virtual-ip-but-no-haproxy >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io > > >
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Am 3/25/25 um 18:55 schrieb Eugen Block:
Okay, so I don't see anything in the keepalive log about communicating between each other. The config files are almost identical, no difference in priority, but in unicast_peer. ceph03 has no entry at all for unicast_peer, ceph02 has only ceph03 in there while ceph01 has both of the others in its unicast_peer entry. That's weird, isn't it?
They should each have the other two as unicast_peers. There must have been a glitch in the service generation. Maybe you should try to remove it and deploy it as new? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Thanks, I removed the ingress service and redeployed it again, with the same result. The interesting part here is, the configs are identical compared to the previous deployment, so the same peers (or no peers) as before. Zitat von Robert Sander <r.sander@heinlein-support.de>:
Am 3/25/25 um 18:55 schrieb Eugen Block:
Okay, so I don't see anything in the keepalive log about communicating between each other. The config files are almost identical, no difference in priority, but in unicast_peer. ceph03 has no entry at all for unicast_peer, ceph02 has only ceph03 in there while ceph01 has both of the others in its unicast_peer entry. That's weird, isn't it?
They should each have the other two as unicast_peers. There must have been a glitch in the service generation. Maybe you should try to remove it and deploy it as new?
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I tried something else, but the result is not really satifying. I edited the keepalive.conf files which had no peers at all or only one peer, so they were all identical. Restarting the daemons helped having only one virtual ip assigned, so now the daemons did communicate and I see messages like these: Master received advert from 192.168.168.112 with same priority 80 but higher IP address than ours Entering BACKUP STATE So that's good. But powering off the machine with the active nfs daemon doesn't provide the expected result. Although keepalive assigns the virtual ip to a different host, the failed nfs daemon lands on the third node, so mounting is not possible. To prevent that from happening, I reduced the number of hosts for nfs and ingress to two. And that seems to work as expected (after modifying the keepalive.conf again). But all in all, the keepalive_only option seems a bit too much manual work at this point. And just a side note: I don't see that a client is connected although I am writing data into the nfs export. Both the dashboard and CLI show no client: ceph nfs export info ebl-nfs-cephfs /nfsovercephfs { "access_type": "RW", "clients": [], "cluster_id": "ebl-nfs-cephfs", ... I only see the active nfs daemon as a CephFS client. Zitat von Eugen Block <eblock@nde.ag>:
Thanks, I removed the ingress service and redeployed it again, with the same result. The interesting part here is, the configs are identical compared to the previous deployment, so the same peers (or no peers) as before.
Zitat von Robert Sander <r.sander@heinlein-support.de>:
Am 3/25/25 um 18:55 schrieb Eugen Block:
Okay, so I don't see anything in the keepalive log about communicating between each other. The config files are almost identical, no difference in priority, but in unicast_peer. ceph03 has no entry at all for unicast_peer, ceph02 has only ceph03 in there while ceph01 has both of the others in its unicast_peer entry. That's weird, isn't it?
They should each have the other two as unicast_peers. There must have been a glitch in the service generation. Maybe you should try to remove it and deploy it as new?
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Adam King
-
Devin A. Bougie
-
Eugen Block
-
Malte Stroem
-
Robert Sander