Here are the yaml files I used to create the NFS and ingress services: nfs-ingress.yaml service_type: ingress service_id: nfs.xcpnfs placement:   count: 2 spec:   backend_service: nfs.xcpnfs   frontend_port: 2049   monitor_port: 9000   virtual_ip: 172.16.172.199/24 nfs.yaml service_type: nfs service_id: xcpnfs placement:   hosts:     - san1     - san2 spec:   port: 20490 Am I missing something here? Is there another mailing list where I should be asking about this? On 31/08/2023 10:38 am, Thorne Lawler wrote:

If there isn't any documentation for this yet, can anyone tell me:

 * How do I inspect/change my NFS/haproxy/keepalived configuration?  * What is it supposed to look like? Does someone have a working example?

Thank you.

On 31/08/2023 9:36 am, Thorne Lawler wrote:

...

Sorry everyone,

Is there any more detailed documentation on the high availability NFS functionality in current Ceph?

This is a pretty serious sticking point.

Thank you.

On 30/08/2023 9:33 am, Thorne Lawler wrote:

...

Fellow cephalopods,

I'm trying to get quick, seamless NFS failover happening on my four-node Ceph cluster.

I followed the instructions here: https://docs.ceph.com/en/latest/cephadm/services/nfs/#high-availability-nfs

but testing shows that failover doesn't happen. When I placed node 2 ("san2") in maintenance mode, the NFS service shut down:

Aug 24 14:19:03 san2 ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq[1962479]: 24/08/2023 04:19:03 : epoch 64b8af5a : san2 : ganesha.nfsd-8[Admin] do_shutdown :MAIN :EVENT :Removing all exports. Aug 24 14:19:13 san2 bash[3235994]: time="2023-08-24T14:19:13+10:00" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq in 10 seconds, resorting to SIGKILL" Aug 24 14:19:13 san2 bash[3235994]: ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq Aug 24 14:19:13 san2 systemd[1]:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic <mailto:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic>e: Main process exited, code=exited, status=137/n/a Aug 24 14:19:14 san2 systemd[1]:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic <mailto:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic>e: Failed with result 'exit-code'. Aug 24 14:19:14 san2 systemd[1]: Stopped Ceph nfs.xcpnfs.1.0.san2.datsvq for e2f1b934-ed43-11ec-80fa-04421a1a1d66.

And that's it. The ingress IP didn't move.

More odd, the cluster seems to have placed the ingress IP on node 1 (san1) but seems to be using the NFS service on node 2.

Do I need to more tightly connect the NFS service to the keepalive and haproxy services, or do I need to expand the ingress services to refer to multiple NFS services?

Thank you.

--

Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

John, Thanks for getting back to me. I am indeed using cephadm, and I will dig up those configurations. Even if Ceph Quincy is current completely incapable of configuring its own HA failover, I would really like to know what the /imagined/ process would be for detecting a node failure and failing over. Can you elaborate about those changes that need to happen, or point me to the forums (ideally the posts_ where this work is broken down in more detail? It's too late for me to change this; I am 100% committed to using Ceph for HA NFS, no matter what that involves. Thanks. On 31/08/2023 11:18 pm, John Mulligan wrote:

On Wednesday, August 30, 2023 8:38:21 PM EDT Thorne Lawler wrote:

...

If there isn't any documentation for this yet, can anyone tell me:

* How do I inspect/change my NFS/haproxy/keepalived configuration? * What is it supposed to look like? Does someone have a working example? The configuration for haproxy, keepalive, and ganesha are generated. I'm assuming you are using cephadm orchestration. If you want to see what it generated configs contain look in /var/lib/ceph/<FSID>/<svc-name>/ under those dirs may be additional subdirectories like etc/ or config/ (it varies from service to service)

It's not simple to customize those files directly. It's not impossible but it's probably not worth it (IMO).

Also, speaking only for my personal opinion, the current NFS situation is one of OK scale-out but is HA in name only. Fail over is not mature. To make it so, changes need to happen throughout the stack including in nfs-ganesha. I know there are some conversations happening around this topic but I don't know the best place to get involved upstream.

I know this probably isn't very satisfactory, but I hope the information helps.

_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io --

Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

Dear list members,, after upgrading to reef (18.2.0) I spent some time with CephFS, NFS & HA(Ingress). I can confirm that Ingress (count either 1 or 2) works well IF only ONE backend server is configured. But this is, of course, no HA. ;-) Two or more backend servers won't work because there isn't ANY "*check*" directive used in HAproxy backend config. This means any client will stay/stick on the last NFS backend server used. IMHO in haproxy.cfg this should look more like the following: [..] backend backend mode tcp option tcp-check tcp-check connect port 2049 balance source hash-type consistent server nfs.nfstest-st.0 10.100.1.111:2049 check server nfs.nfstest-st.1 10.100.1.112:2049 check BTW: There are plenty of configuration directives to get a proper working NFS backend. Please refer to: https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#5.2-check https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#stick-table https://infohub.delltechnologies.com/l/ecs-with-haproxy-load-balancer-2/nfs-... Did (NFS) HA ever work in previous versions? Unfortunately, I had no reason to test this in advance. Currently, only a manual installation of HAproxy based ingress controller(s) should help. And, can we hope for a correction in future versions? Thanks, Christoph Am Mo., 4. Sept. 2023 um 07:43 Uhr schrieb Thorne Lawler <thorne@ddns.com.au

:

One more question, for John or anyone:

It looks like Ganesha NFS wants to use dbus to talk to other services (although I'm not sure how well this works across containers).

I just realised that my build (on AlmaLinux 8 'minimal') did not include dbus, and Ceph has not installed it.

I have manually installed the default 'dbus' package and its deps, but this doesn't seem to have had any effect.

* Does Ganesha use dbus to talk to other services, even in a cephadm containerised build? * Is there a non-destructive way to make Ganesha retry gsh_dbus_pkginit ? * Is the default dbus package sufficient, or does Ceph require specific dbus plugins?

Thank you.

On 4/09/2023 9:55 am, Thorne Lawler wrote:

...

John,

Thanks for getting back to me. I am indeed using cephadm, and I will dig up those configurations.

Even if Ceph Quincy is current completely incapable of configuring its own HA failover, I would really like to know what the /imagined/ process would be for detecting a node failure and failing over.

Can you elaborate about those changes that need to happen, or point me to the forums (ideally the posts_ where this work is broken down in more detail?

It's too late for me to change this; I am 100% committed to using Ceph for HA NFS, no matter what that involves.

Thanks.

On 31/08/2023 11:18 pm, John Mulligan wrote:

...

On Wednesday, August 30, 2023 8:38:21 PM EDT Thorne Lawler wrote:

...

If there isn't any documentation for this yet, can anyone tell me:

* How do I inspect/change my NFS/haproxy/keepalived configuration? * What is it supposed to look like? Does someone have a working example? The configuration for haproxy, keepalive, and ganesha are generated. I'm assuming you are using cephadm orchestration. If you want to see what it generated configs contain look in /var/lib/ceph/<FSID>/<svc-name>/ under those dirs may be additional subdirectories like etc/ or config/ (it varies from service to service)

It's not simple to customize those files directly. It's not impossible but it's probably not worth it (IMO).

Also, speaking only for my personal opinion, the current NFS situation is one of OK scale-out but is HA in name only. Fail over is not mature. To make it so, changes need to happen throughout the stack including in nfs-ganesha. I know there are some conversations happening around this topic but I don't know the best place to get involved upstream.

I know this probably isn't very satisfactory, but I hope the information helps.

_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io --

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. / _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

On Tuesday, September 26, 2023 6:00:23 AM EDT Ackermann, Christoph wrote:

Dear list members,,

after upgrading to reef (18.2.0) I spent some time with CephFS, NFS & HA(Ingress). I can confirm that Ingress (count either 1 or 2) works well IF only ONE backend server is configured. But this is, of course, no HA. ;-) Two or more backend servers won't work because there isn't ANY "*check*" directive used in HAproxy backend config. This means any client will stay/stick on the last NFS backend server used. IMHO in haproxy.cfg this should look more like the following:

[..] backend backend mode tcp option tcp-check tcp-check connect port 2049 balance source hash-type consistent server nfs.nfstest-st.0 10.100.1.111:2049 check server nfs.nfstest-st.1 10.100.1.112:2049 check

BTW: There are plenty of configuration directives to get a proper working NFS backend. Please refer to:

https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#5.2-check

https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#stick-table

https://infohub.delltechnologies.com/l/ecs-with-haproxy-load-balancer-2/nfs-> configuration-definitions/

Did (NFS) HA ever work in previous versions? Unfortunately, I had no reason to test this in advance. Currently, only a manual installation of HAproxy based ingress controller(s) should help.

And, can we hope for a correction in future versions?

FWIW there is a tracker for this specific issue https://tracker.ceph.com/ issues/62638 I don't think that alone will make a full HA solution but it would be an improvement. :-)

Thanks, Christoph

Am Mo., 4. Sept. 2023 um 07:43 Uhr schrieb Thorne Lawler <thorne@ddns.com.au

...

One more question, for John or anyone:

It looks like Ganesha NFS wants to use dbus to talk to other services (although I'm not sure how well this works across containers).

I just realised that my build (on AlmaLinux 8 'minimal') did not include dbus, and Ceph has not installed it.

I have manually installed the default 'dbus' package and its deps, but this doesn't seem to have had any effect.

* Does Ganesha use dbus to talk to other services, even in a cephadm

containerised build?

* Is there a non-destructive way to make Ganesha retry gsh_dbus_pkginit ? * Is the default dbus package sufficient, or does Ceph require

specific dbus plugins?

Thank you.

On 4/09/2023 9:55 am, Thorne Lawler wrote:

...

John,

Thanks for getting back to me. I am indeed using cephadm, and I will dig up those configurations.

Even if Ceph Quincy is current completely incapable of configuring its own HA failover, I would really like to know what the /imagined/ process would be for detecting a node failure and failing over.

Can you elaborate about those changes that need to happen, or point me to the forums (ideally the posts_ where this work is broken down in more detail?

It's too late for me to change this; I am 100% committed to using Ceph for HA NFS, no matter what that involves.

Thanks.

On 31/08/2023 11:18 pm, John Mulligan wrote:

...

On Wednesday, August 30, 2023 8:38:21 PM EDT Thorne Lawler wrote:

...

If there isn't any documentation for this yet, can anyone tell me: * How do I inspect/change my NFS/haproxy/keepalived configuration? * What is it supposed to look like? Does someone have a working

example?

The configuration for haproxy, keepalive, and ganesha are generated. I'm assuming you are using cephadm orchestration. If you want to see what it generated configs contain look in /var/lib/ceph/<FSID>/<svc-name>/ under those dirs may be additional subdirectories like etc/ or config/ (it varies from service to service)

It's not simple to customize those files directly. It's not impossible but it's probably not worth it (IMO).

Also, speaking only for my personal opinion, the current NFS situation is one of OK scale-out but is HA in name only. Fail over is not mature. To make it so, changes need to happen throughout the stack including in nfs-ganesha. I know there are some conversations happening around this topic but I don't know the best place to get involved upstream.

I know this probably isn't very satisfactory, but I hope the information helps.

_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io

--

Regards,

Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. / _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io