MGR does not generate prometheus config for ceph-exporter
Hello, I have issue with multiple of our clusters. It doesn't seem to be versions specific and just happens on our clusters which have been running for a while and have been through some version upgrades. In current cephadm versions prometheus is configured via service discovery from MGR. Relevant prometheus config looks like this: - job_name: 'ceph-exporter' honor_labels: true http_sd_configs: - url: http://X.X.X.X:8765/sd/prometheus/sd-config?service=ceph-exporter Issue is that mgr doesn't return any hosts. Just an empty array []. Prometheus doesn't scrap ceph-exporter and some of our grafana graphs are empty. :/ Other URLs in this config file work fine and generate valid sd configs. Has anyone else experienced this? Does anyone know how to fix this? Best regards Adam Prycki
Hi, I haven't had an issue yet, but the first question would be if you failed the mgr. ;-) The second question is, does the ceph-exporter respond? ceph01:~ # curl http://IP:9926/metrics Does the ceph-exporter log anything? Do you have any custom configs or templates in place? ceph01:~ # ceph config-key get mgr/cephadm/services/prometheus/prometheus.yml Regards, Eugen Zitat von Adam Prycki <aprycki@man.poznan.pl>:
Hello,
I have issue with multiple of our clusters. It doesn't seem to be versions specific and just happens on our clusters which have been running for a while and have been through some version upgrades.
In current cephadm versions prometheus is configured via service discovery from MGR.
Relevant prometheus config looks like this: - job_name: 'ceph-exporter' honor_labels: true http_sd_configs: - url: http://X.X.X.X:8765/sd/prometheus/sd-config?service=ceph-exporter Issue is that mgr doesn't return any hosts. Just an empty array []. Prometheus doesn't scrap ceph-exporter and some of our grafana graphs are empty. :/
Other URLs in this config file work fine and generate valid sd configs.
Has anyone else experienced this? Does anyone know how to fix this?
Best regards Adam Prycki
participants (2)
-
Adam Prycki
-
Eugen Block