Re: [mgmt-gateway] monitoring problems, origin not allowed in grafana
Hi Gilles, I was able to reproduce a similar problem locally (the error related to "origin not allowed). As of my local testing (and observations) Grafana was failing with “origin not allowed” only when user login was enabled (user/password), but everything worked fine when using anonymous access. I think this difference is due to how Grafana’s CSRF protection works: With anonymous access: - You are effectively a non-authenticated user. - Most requests are simple GETs and there is no session cookie to protect. - Grafana’s CSRF checks are much less strict, so it doesn’t really care about the Origin header. With user login enabled: - After logging in, the browser sends requests with a session cookie and an Origin header like: Origin: https://<your-fqdn-domain> - Because this is now an authenticated session, Grafana enforces strict CSRF checks. - If the Origin does not exactly match what Grafana considers its own URL or trusted origins (Grafana is configured with an internal hostname, while you access it via the mgmt-gateway FQDN), Grafana rejects the request with: “origin not allowed”. I think that explains why the problem only appears once user authentication is enabled. The problem is that Grafana is deployed in this case behind mgmt-gateway (which acts as reverse-proxy). So Grafana has no idea what's the fqdn in use as it comes from the mgmt-gateway. When oauth2-proxy is also used so authentication goes through SSO this issue doesn't happen. To solve the problem, grafana provides a config field csrf_trusted_origins which can be used to indicate manually your trusted origins. In this case, you would need to add something like: [security] csrf_trusted_origins = <your-fqdn-domain> For example: [security] csrf_trusted_origins = my-cloud.ceph-node-0.com In your grafana.ini (located at /var/lib/ceph/<fsid>/grafana.<your-grafana-node>/etc/grafana/grafana.ini To test this quickly you can edit .ini manually and then run "ceph orch restart grafana" from the cephadm shell. This change is unfortunately not permanent, if you redeploy your grafana or move it to another node, you will lose the change. To make it permanent you have to adjust the grafana Jinja template, used by cephadm to generate the ini file. For that you have to modify the template to add the above entry and store it for example in a grafana.ini.j2 file then run the following commands: ceph config-key set mgr/cephadm/services/grafana/grafana.ini -i grafana.ini.j2 ceph orch reconfig grafana To get the template you can either find it in github, for tentacle (plz adjust the path to your exact release): https://github.com/ceph/ceph/blob/tentacle/src/pybind/mgr/cephadm/templates/... or directly from the ceph-mgr container: /usr/share/ceph/mgr/cephadm/templates/services/grafana/grafana.ini.j2 Best, Redouane. On Thu, Dec 11, 2025 at 9:27 AM Redouane Kachach via ceph-users < ceph-users@ceph.io> wrote:
Thank you Gilles for updating the ticket with the info,
One last thing that could shed some light is to see the cert used by Prometheus (then let's continue the discussion on the tracker),
openssl x509 -in /var/lib/ceph/<fsid>/prometheus.<host>/etc/prometheus/prometheus.crt -text
(Plz omit any sensitive information for your lab from the output)
BTW: there's a Slack channel dedicated for cephadm that can be used for a more interactive dicussion https://ceph-storage.slack.com/archives/C04SNUBD2M6
Best, Redouane.
On Wed, Dec 10, 2025 at 5:06 PM Gilles Mocellin < gilles.mocellin@nuagelibre.org> wrote:
Thank you Redouane,
I have updated the ticket with all what you made me do and what you asked. Except that "--include-cephadm-signed" does not exist on my version for `ceph orch certmgr cert ls`.
https://tracker.ceph.com/issues/74164
Le 2025-12-10 15:36, Redouane Kachach a écrit :
Thanks for the feedback Gilles.
Using your own cert for the mgmt-gateway is Okay and shouldn't be an issue. Errors arise when you use custom certs for the Dashboard itself as bcz when the mgmt-gateway is enabled mTLS is enforced for internal communications (Dashboard <-> Prometheus and Alertmanager <-> Dashbaord i.e)
From what you are commenting it seems like Grafana is not able to get metrics from Prometheus.
Can you plz update the ticket with the output of the command (obviously omitting any private/sensitive data):
ceph orch certmgr cert ls --include-cephadm-signed
This would show all the certificates in use for the different services.
It may also help see the config of Grafana:
cat /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/grafana.ini cat
/var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/provisioning/datasources/ceph-dashboard.yml
Thanks, Redouane.
On Wed, Dec 10, 2025 at 2:33 PM Gilles Mocellin < gilles.mocellin@nuagelibre.org> wrote:
Better but not completely.
I don't have errors anymore concerning alertmanager, and I can see alerts, thereis on concerning mgr/prometheus module not responding. Even if I disable/enable it.
In the MGR logs, I see certificate problem :
Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01
[10/Dec/2025:13:27:40] ENGINE Client ('::ffff:100.70.100.51', 55206, 0, 0) lost — peer dropped the TLS connection suddenly, during handshake: (1, '[SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate (_ssl.c:1147)') Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 ceph-mgr[7789]: [prometheus INFO cherrypy.error] [10/Dec/2025:13:27:40] ENGINE Client ('::ffff:100.70.100.51', 55206, 0, 0) lost — peer dropped the TLS connection suddenly, during handshake: (1, '[SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate (_ssl.c:1147)')
Grafana Dashboard still don't work. In embeded mode, I still have Unauthorized messages. If I open directly Grafana, with the DNS name, I can list the Dashboard, but in them, there is still "origin not allowed" messages. I cannot log in, with also the message "origin not allowed".
Le 2025-12-10 14:15, Gilles Mocellin a écrit :
Hi Redouane,
Yes, I'm using my own certificates, signed by our internal PKI.
I will try, but I still want a valid certificate for my DNS entry, certificate guven to the mgmt-gateway service.
Le 10 décembre 2025 13:28:45 GMT+01:00, Redouane Kachach <rkachach@redhat.com> a écrit :
Hi Gilles,
What certificate are you using on your Dashboard?
Just to rule out any certificate issues: when using mgmt-gateway,
dashboard must be configured to use cephadm-signed certificates. This is normally done automatically, but if you were using custom certificates for your dashboard, those will not be overwritten by cephadm. If this is the case then plz just follow this procedure to fix
1) generate cephadm-signed certs for the dashboard (from cephadm shell):
ceph orch certmgr generate-certificates dashboard \ | tee \ >(jq -r '.cert' > dashboard.cert.pem) \ >(jq -r '.key' > dashboard.key.pem) \ > /dev/null
2) The above will generate two files: dashboard.cert.pem and dashboard.key.pem, now we have to use them to set the dashboard cert/key:
ceph dashboard set-ssl-certificate -i dashboard.cert.pem ceph dashboard set-ssl-certificate-key -i dashboard.key.pem
3) Reload the dashboard, so it starts using the new populated certificates:
ceph mgr module disable dashboard; ceph mgr module enable
dashboard
--force
Best, Redouane.
On Wed, Dec 10, 2025 at 11:31 AM Gilles Mocellin via ceph-users < ceph-users@ceph.io> wrote:
> Hello ! > > Does someone manage to use mgmt-gateway, with a FQDN, and the > monitoring > stack working ? > > In my LAB, after upgrading from 10.2.3 to 20.2.0, > My dashboard was working fine, with grafana dashboards. > > I decided to test mgmt-gateway. > I use a VIP with a specific DNS name pointing at that VIP. > I have a certificate generated with our internal PKI, valid for
> DNS > name and the VIP. > > Dashboard works, but no monitoring anymore. > - Popups concerning alermanager : > > Failed to execute prometheus > Error parsing Prometheus Alertmanager response: Expecting value > 9/12/25 04:07 PM > > - No graphs in dashboard, with errors "origin not allowed" in each > panel > > If I open Grafana directly, I see : > - that the VIP is used, not my DNS name, but none works. > - Grafana works but I can't see dashboards, and if I put a URL
ceph-8ec7575a-7de5-11f0-a78a-246e96bd90a4-mgr-fidcl-lyo1-sto-sds-lab-01-gtcifn[7494]: the them: the that
> should work (https://$VIP/grafana/d/edtb0oxdq/ceph-cluster), I have a > message : > > Failed to load dashboard > Unauthorized > > Some more infos : > > https://$DNS/grafana/dashboards => show the list of Dashboards > Open a dashboard show errors 'origin not allowed" on panels > > https://$VIP/grafana/dashboards => cannot list the Dashboards, error > Unauthorized > > Concerning the Altermanger errors, I see thoses XHR calls in my > browser, > ending in HTTP error 400 Bad Request : > https://$DNS/api/prometheus?cluster_filter=true > > I opened an issue here : https://tracker.ceph.com/issues/74164 > > -- > Gilles > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io > >
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 2025-12-11 12:06, Redouane Kachach a écrit :
Hi Gilles,
I was able to reproduce a similar problem locally (the error related to "origin not allowed). As of my local testing (and observations) Grafana was failing with “origin not allowed” only when user login was enabled (user/password), but everything worked fine when using anonymous access. I think this difference is due to how Grafana’s CSRF protection works:
[...] You are right !
I think that explains why the problem only appears once user authentication is enabled. The problem is that Grafana is deployed in this case behind mgmt-gateway (which acts as reverse-proxy). So Grafana has no idea what's the fqdn in use as it comes from the mgmt-gateway. When oauth2-proxy is also used so authentication goes through SSO this issue doesn't happen.
Ah ! As I want to also test that, perhaps that will solve my problem.
To solve the problem, grafana provides a config field csrf_trusted_origins which can be used to indicate manually your trusted origins. In this case, you would need to add something like:
[security] csrf_trusted_origins = <your-fqdn-domain> [...]
This works for direct access. But to work also in the Dashboards, I need to define the grafana frontend URL, else the VIP is used to call Grafana. $ ceph dashboard set-grafana-frontend-api-url https://$DNSforVIP/grafana
This change is unfortunately not permanent, if you redeploy your grafana or move it to another node, you will lose the change. To make it permanent you have to adjust the grafana Jinja template, used by cephadm to generate the ini file. For that you have to modify the template to add the above entry and store it for example in a grafana.ini.j2 file then run the following commands:
I'm not fan of modifying that myself. So I will wait if it comes upstream. OR, as I mention it the issue (https://tracker.ceph.com/issues/74164), We don't need to log in if a Dashboard can show us logs. There are several on Grafana dashboards Web site. For now, I will logout from Grafana, and if I want to access logs with Grafana/Loki, I will use the node's IP directly.
Best, Redouane.
Thanks a lot to point the problems !
participants (2)
-
Gilles Mocellin
-
Redouane Kachach