Grafana, Prometheus, client certificates and mgmt-gateway in 20.2.3
Hi, we stumbled across an issue that is introduced with the new mgmt-gateway service. As soon as it is active the Prometheus service is configured to require client certificates for its API. It looks like the Grafana service is configured with client certificates according to the new service spec but it does not seem to use them. The result is no metrics in the Grafana dashboards. Can anyone confirm this behaviour? After removing the mgmt-gateway service and redeploying all monitoring services everything works again. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi Robert, When mgmt-gateway is deployed, direct external access to Grafana/Prometheus is intentionally disabled. The monitoring services are expected to be accessed/routed through mgmt-gateway. That's the whole point behind the mgmt-gateway design. Client certs are required bcz when the mgmt-gateway is deployed strict mTLS is enforced for service-to-service communication. Regards, Redo. On Mon, Sep 14, 2026 at 5:22 PM Robert Sander <ceph-users@ceph.io> wrote:
Hi,
we stumbled across an issue that is introduced with the new mgmt-gateway service. As soon as it is active the Prometheus service is configured to require client certificates for its API.
It looks like the Grafana service is configured with client certificates according to the new service spec but it does not seem to use them.
The result is no metrics in the Grafana dashboards.
Can anyone confirm this behaviour?
After removing the mgmt-gateway service and redeploying all monitoring services everything works again.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin <https://www.google.com/maps/search/Schwedter+Str.+8%2F9b,+10119+Berlin?entry=gmail&source=g>
https://www.heinlein-support.de
Tel: +49 30 405051 <+49%2030%20405051> - 0 Fax: +49 30 405051 - 19 <+49%2030%2040505119>
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Redouane, Am 15.09.26 um 12:43 PM schrieb Redouane Kachach:
When mgmt-gateway is deployed, direct external access to Grafana/ Prometheus is intentionally disabled. The monitoring services are expected to be accessed/routed through mgmt-gateway. That's the whole point behind the mgmt-gateway design. Client certs are required bcz when the mgmt-gateway is deployed strict mTLS is enforced for service-to-service communication.
I understand. We did not try to access them directly but through the service IP of the mgmt-gateway service. Grafana was not displaying the metrics and when investigating further we found timeouts that pointed to mTLS client certificates. I will try to reproduce the setup in a test cluster. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
I see, another important point to check is what URL you are using to access the service. Specifically, are you accessing it using the service IP, a short hostname, or an FQDN? Grafana can be sensitive to the hostname used because cephadm configures its domain/root_url for the mgmt-gateway setup. If you access it using an IP while Grafana is configured for a hostname/FQDN, the Grafana panels may not work correctly even though the backend services are healthy. You can see the URL configured for grafana by running:
ceph dashboard get-grafana-api-url
On Tue, Sep 15, 2026 at 1:08 PM Robert Sander <r.sander@heinlein-support.de> wrote:
Hi Redouane,
Am 15.09.26 um 12:43 PM schrieb Redouane Kachach:
When mgmt-gateway is deployed, direct external access to Grafana/ Prometheus is intentionally disabled. The monitoring services are expected to be accessed/routed through mgmt-gateway. That's the whole point behind the mgmt-gateway design. Client certs are required bcz when the mgmt-gateway is deployed strict mTLS is enforced for service-to-service communication.
I understand. We did not try to access them directly but through the service IP of the mgmt-gateway service.
Grafana was not displaying the metrics and when investigating further we found timeouts that pointed to mTLS client certificates.
I will try to reproduce the setup in a test cluster.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 <+49%2030%20405051> - 0 Fax: +49 30 405051 - 19 <+49%2030%2040505119>
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi, Am 15.09.26 um 1:35 PM schrieb Redouane Kachach:
I see, another important point to check is what URL you are using to access the service. Specifically, are you accessing it using the service IP, a short hostname, or an FQDN?
Grafana can be sensitive to the hostname used because cephadm configures its domain/root_url for the mgmt-gateway setup. If you access it using an IP while Grafana is configured for a hostname/FQDN, the Grafana panels may not work correctly even though the backend services are healthy.
The URLs all contain the service IP and no hostname as there was no DNS available. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
In case accessing via URL using an IP fails, can you please try using the hostname? (short hostname in case no fqdn) On Tue, Sep 15, 2026 at 3:17 PM Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
Am 15.09.26 um 1:35 PM schrieb Redouane Kachach:
I see, another important point to check is what URL you are using to access the service. Specifically, are you accessing it using the service IP, a short hostname, or an FQDN?
Grafana can be sensitive to the hostname used because cephadm configures its domain/root_url for the mgmt-gateway setup. If you access it using an IP while Grafana is configured for a hostname/FQDN, the Grafana panels may not work correctly even though the backend services are healthy.
The URLs all contain the service IP and no hostname as there was no DNS available.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 <+49%2030%20405051> - 0 Fax: +49 30 405051 - 19 <+49%2030%2040505119>
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi, we have now removed the mgmt-gateway service. We came to the conclusion that it is not production ready yet. The certificates do not match. I am using this service spec: service_type: mgmt-gateway service_name: mgmt-gateway placement: label: mgmt-gateway spec: port: 443 virtual_ip: ceph.example.com ssl: true ssl_cert: |- -----BEGIN CERTIFICATE----- XXXX -----END CERTIFICATE----- ssl_key: |- -----BEGIN PRIVATE KEY----- XXXX -----END PRIVATE KEY----- --- service_type: ingress service_id: mgmt-gateway placement: label: mgmt-gateway virtual_ip: 10.128.8.255/22 backend_service: mgmt-gateway keepalive_only: true ceph.example.com has an A record that resolves to 10.128.8.255. The provided certificate contains a subjectAlternativeName for ceph.example.com. The browser is able to open https://ceph.example.com ceph dashboard set-grafana-frontend-api-url https://ceph.example.com/grafana/ The browser is able to open Grafana. But Grafana is not able to reach Prometheus. mgr/dashboard/PROMETHEUS_API_HOST is set to https://ceph.example.com:29443/internal/prometheus which works but the certificate at this port does not. Bu the nginx gets an internal certificate configured on Port 29443 that was signed by the certmgr's internal CA and contains the hostname where the daemon is deployed in the subjectAlternativeName attribute. It should have ceph.example.com in subjectAlternativeName as this is the name for the service IP and is set in PROMETHEUS_API_HOST. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
I see, I think the problem is the virtual_ip field which has an incorrect value: virtual_ip: ceph.example.com (INCORRECT) This field has to be set to the virtual_ip, and that's explains why you end up with this entry in the internal PROMETHEUS_API_HOST. The config has to be: =========================== service_type: mgmt-gateway spec: virtual_ip: 10.128.8.255 ssl: true ssl_cert: ... ssl_key: ... --- service_type: ingress service_id: mgmt-gateway virtual_ip: 10.128.8.255/22 backend_service: mgmt-gateway keepalive_only: true =========================== As explained in the docs: https://docs.ceph.com/en/latest/cephadm/services/mgmt-gateway/ Note: Just to confirm the "/22" in the ingress part is correct and you have to keep it otherwise ingress wont work correctly On Fri, Sep 18, 2026 at 11:09 AM Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
we have now removed the mgmt-gateway service.
We came to the conclusion that it is not production ready yet. The certificates do not match.
I am using this service spec:
service_type: mgmt-gateway service_name: mgmt-gateway placement: label: mgmt-gateway spec: port: 443 virtual_ip: ceph.example.com ssl: true ssl_cert: |- -----BEGIN CERTIFICATE----- XXXX -----END CERTIFICATE----- ssl_key: |- -----BEGIN PRIVATE KEY----- XXXX -----END PRIVATE KEY----- --- service_type: ingress service_id: mgmt-gateway placement: label: mgmt-gateway virtual_ip: 10.128.8.255/22 backend_service: mgmt-gateway keepalive_only: true
ceph.example.com has an A record that resolves to 10.128.8.255. The provided certificate contains a subjectAlternativeName for ceph.example.com.
The browser is able to open https://ceph.example.com
ceph dashboard set-grafana-frontend-api-url https://ceph.example.com/grafana/
The browser is able to open Grafana.
But Grafana is not able to reach Prometheus.
mgr/dashboard/PROMETHEUS_API_HOST is set to https://ceph.example.com:29443/internal/prometheus which works but the certificate at this port does not.
Bu the nginx gets an internal certificate configured on Port 29443 that was signed by the certmgr's internal CA and contains the hostname where the daemon is deployed in the subjectAlternativeName attribute.
It should have ceph.example.com in subjectAlternativeName as this is the name for the service IP and is set in PROMETHEUS_API_HOST.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin <https://www.google.com/maps/search/Schwedter+Str.+8%2F9b,+10119+Berlin?entry=gmail&source=g>
https://www.heinlein-support.de
Tel: +49 30 405051 <+49%2030%20405051> - 0 Fax: +49 30 405051 - 19 <+49%2030%2040505119>
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi, Am 18.09.26 um 1:58 PM schrieb Redouane Kachach:
I see, I think the problem is the virtual_ip field which has an incorrect value:
virtual_ip: ceph.example.com <http://ceph.example.com/> (INCORRECT)
This field has to be set to the virtual_ip, and that's explains why you end up with this entry in the internal PROMETHEUS_API_HOST.
And then I am on square one again. Prometheus requires a client certificate for authentication, Grafana does not send one: # cat etc/prometheus/web.yml tls_server_config: cert_file: prometheus.crt key_file: prometheus.key client_auth_type: RequireAndVerifyClientCert client_ca_file: root_cert.pem It looks like a client certificate is configured in etc/grafana/provisioning/datasources/ceph-dashboard.yml but the Grafana dashboard does not display any metrics. A yellow notification box pops up with the text: <html> <head><title>400 No required SSL certificate was sent</title></head> <body> <center><h1>400 Bad Request</h1></center> <center>No required SSL certificate was sent</center> <hr><center>nginx/1.26.1</center> </body> </html> Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
How are you trying to access grafana? Through the Ceph dashboard or by trying to access its URL directly? When mgmt-gatewa is enabled mTLS is enforced and the only allowed access is through the Ceph Dashboard. Note: Can you plz open a tracker and share your Grafana config? This way is easier for devs to help with the issues you are facing. https://tracker.ceph.com/ On Fri, Sep 18, 2026 at 2:47 PM Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
Am 18.09.26 um 1:58 PM schrieb Redouane Kachach:
I see, I think the problem is the virtual_ip field which has an incorrect value:
virtual_ip: ceph.example.com <http://ceph.example.com/> (INCORRECT)
This field has to be set to the virtual_ip, and that's explains why you end up with this entry in the internal PROMETHEUS_API_HOST.
And then I am on square one again.
Prometheus requires a client certificate for authentication, Grafana does not send one:
# cat etc/prometheus/web.yml tls_server_config: cert_file: prometheus.crt key_file: prometheus.key client_auth_type: RequireAndVerifyClientCert client_ca_file: root_cert.pem
It looks like a client certificate is configured in etc/grafana/provisioning/datasources/ceph-dashboard.yml but the Grafana dashboard does not display any metrics.
A yellow notification box pops up with the text:
<html> <head><title>400 No required SSL certificate was sent</title></head> <body> <center><h1>400 Bad Request</h1></center> <center>No required SSL certificate was sent</center> <hr><center>nginx/1.26.1</center> </body> </html>
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 <+49%2030%20405051> - 0 Fax: +49 30 405051 - 19 <+49%2030%2040505119>
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Am 18.09.26 um 7:39 PM schrieb Redouane Kachach:
How are you trying to access grafana? Through the Ceph dashboard or by trying to access its URL directly?
Through the Ceph dashboard.
Note: Can you plz open a tracker and share your Grafana config? This way is easier for devs to help with the issues you are facing.
I did create this issue: https://tracker.ceph.com/issues/80983 Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Thanks Robert, we will try to reproduce the issue and update the tracker. On Mon, Sep 28, 2026 at 3:55 PM Robert Sander <r.sander@heinlein-support.de> wrote:
Am 18.09.26 um 7:39 PM schrieb Redouane Kachach:
How are you trying to access grafana? Through the Ceph dashboard or by trying to access its URL directly?
Through the Ceph dashboard.
Note: Can you plz open a tracker and share your Grafana config? This way is easier for devs to help with the issues you are facing.
I did create this issue: https://tracker.ceph.com/issues/80983
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 <+49%2030%20405051> - 0 Fax: +49 30 405051 - 19 <+49%2030%2040505119>
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
participants (2)
-
Redouane Kachach
-
Robert Sander