Re: [mgmt-gateway] monitoring problems, origin not allowed in grafana
Thank you Gilles for updating the ticket with the info, One last thing that could shed some light is to see the cert used by Prometheus (then let's continue the discussion on the tracker), openssl x509 -in /var/lib/ceph/<fsid>/prometheus.<host>/etc/prometheus/prometheus.crt -text (Plz omit any sensitive information for your lab from the output) BTW: there's a Slack channel dedicated for cephadm that can be used for a more interactive dicussion https://ceph-storage.slack.com/archives/C04SNUBD2M6 Best, Redouane. On Wed, Dec 10, 2025 at 5:06 PM Gilles Mocellin < gilles.mocellin@nuagelibre.org> wrote:
Thank you Redouane,
I have updated the ticket with all what you made me do and what you asked. Except that "--include-cephadm-signed" does not exist on my version for `ceph orch certmgr cert ls`.
https://tracker.ceph.com/issues/74164
Le 2025-12-10 15:36, Redouane Kachach a écrit :
Thanks for the feedback Gilles.
Using your own cert for the mgmt-gateway is Okay and shouldn't be an issue. Errors arise when you use custom certs for the Dashboard itself as bcz when the mgmt-gateway is enabled mTLS is enforced for internal communications (Dashboard <-> Prometheus and Alertmanager <-> Dashbaord i.e)
From what you are commenting it seems like Grafana is not able to get metrics from Prometheus.
Can you plz update the ticket with the output of the command (obviously omitting any private/sensitive data):
ceph orch certmgr cert ls --include-cephadm-signed
This would show all the certificates in use for the different services.
It may also help see the config of Grafana:
cat /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/grafana.ini cat
/var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/provisioning/datasources/ceph-dashboard.yml
Thanks, Redouane.
On Wed, Dec 10, 2025 at 2:33 PM Gilles Mocellin < gilles.mocellin@nuagelibre.org> wrote:
Better but not completely.
I don't have errors anymore concerning alertmanager, and I can see alerts, thereis on concerning mgr/prometheus module not responding. Even if I disable/enable it.
In the MGR logs, I see certificate problem :
Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01
ceph-8ec7575a-7de5-11f0-a78a-246e96bd90a4-mgr-fidcl-lyo1-sto-sds-lab-01-gtcifn[7494]:
[10/Dec/2025:13:27:40] ENGINE Client ('::ffff:100.70.100.51', 55206, 0, 0) lost — peer dropped the TLS connection suddenly, during handshake: (1, '[SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate (_ssl.c:1147)') Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 ceph-mgr[7789]: [prometheus INFO cherrypy.error] [10/Dec/2025:13:27:40] ENGINE Client ('::ffff:100.70.100.51', 55206, 0, 0) lost — peer dropped the TLS connection suddenly, during handshake: (1, '[SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate (_ssl.c:1147)')
Grafana Dashboard still don't work. In embeded mode, I still have Unauthorized messages. If I open directly Grafana, with the DNS name, I can list the Dashboard, but in them, there is still "origin not allowed" messages. I cannot log in, with also the message "origin not allowed".
Le 2025-12-10 14:15, Gilles Mocellin a écrit :
Hi Redouane,
Yes, I'm using my own certificates, signed by our internal PKI.
I will try, but I still want a valid certificate for my DNS entry, certificate guven to the mgmt-gateway service.
Le 10 décembre 2025 13:28:45 GMT+01:00, Redouane Kachach <rkachach@redhat.com> a écrit :
Hi Gilles,
What certificate are you using on your Dashboard?
Just to rule out any certificate issues: when using mgmt-gateway, the dashboard must be configured to use cephadm-signed certificates. This is normally done automatically, but if you were using custom certificates for your dashboard, those will not be overwritten by cephadm. If this is the case then plz just follow this procedure to fix them:
1) generate cephadm-signed certs for the dashboard (from cephadm shell):
ceph orch certmgr generate-certificates dashboard \ | tee \ >(jq -r '.cert' > dashboard.cert.pem) \ >(jq -r '.key' > dashboard.key.pem) \
/dev/null
2) The above will generate two files: dashboard.cert.pem and dashboard.key.pem, now we have to use them to set the dashboard cert/key:
ceph dashboard set-ssl-certificate -i dashboard.cert.pem ceph dashboard set-ssl-certificate-key -i dashboard.key.pem
3) Reload the dashboard, so it starts using the new populated certificates:
ceph mgr module disable dashboard; ceph mgr module enable dashboard --force
Best, Redouane.
On Wed, Dec 10, 2025 at 11:31 AM Gilles Mocellin via ceph-users < ceph-users@ceph.io> wrote:
Hello !
Does someone manage to use mgmt-gateway, with a FQDN, and the monitoring stack working ?
In my LAB, after upgrading from 10.2.3 to 20.2.0, My dashboard was working fine, with grafana dashboards.
I decided to test mgmt-gateway. I use a VIP with a specific DNS name pointing at that VIP. I have a certificate generated with our internal PKI, valid for the DNS name and the VIP.
Dashboard works, but no monitoring anymore. - Popups concerning alermanager :
Failed to execute prometheus Error parsing Prometheus Alertmanager response: Expecting value 9/12/25 04:07 PM
- No graphs in dashboard, with errors "origin not allowed" in each panel
If I open Grafana directly, I see : - that the VIP is used, not my DNS name, but none works. - Grafana works but I can't see dashboards, and if I put a URL that should work (https://$VIP/grafana/d/edtb0oxdq/ceph-cluster), I
have a
message :
Failed to load dashboard Unauthorized
Some more infos :
https://$DNS/grafana/dashboards => show the list of Dashboards Open a dashboard show errors 'origin not allowed" on panels
https://$VIP/grafana/dashboards => cannot list the Dashboards, error Unauthorized
Concerning the Altermanger errors, I see thoses XHR calls in my browser, ending in HTTP error 400 Bad Request : https://$DNS/api/prometheus?cluster_filter=true
I opened an issue here : https://tracker.ceph.com/issues/74164
-- Gilles _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (1)
-
Redouane Kachach