You could check Alertmanager container logs <https://docs.ceph.com/en/quincy/cephadm/operations/#example-of-logging-to-journald> . Kind Regards, Ernesto On Wed, Apr 16, 2025 at 4:54 PM Tim Holloway <timh@mousetech.com> wrote:
I'm thinking more some sort of latency error.
I have 2 prometheus daemons running at the moment. The hosts files on all my ceph servers contain both hostname and FQDN.
This morning the alert was gone. I don't know where I might find a log of when it comes and goes, but all was clean, then it wasn't, now it's clean again and I haven't been playing with any sort of configurations or bouncing hosts or services. It's just appearing and disappearing.
Tim
On 4/16/25 09:34, Ankush Behl wrote:
Just to add upon what Ernesto mentioned. Your prometheus container might not be able to reachout to ceph scrape job as the it could be using FQDN or Hostname. Try updating /etc/hosts with ip and hostname of the ceph scrape job(you can find it on prometheus UI -> status -> targets) and restarting the prometheus after that might help resolve the issue.
On Wed, Apr 16, 2025 at 2:10 PM Ernesto Puerta <epuertat@redhat.com> wrote:
Don't shoot the messenger. Dashboard is just displaying the alert that Prometheus/AlertManager is reporting. The alert definition is here <
https://github.com/ceph/ceph/blob/3993779cde9d10512f4a26f87487d11103ac1bd0/m...
. As you may see, it's based on the status of the Prometheus "ceph" scrape job. This alert is vital, because if the "ceph" job is not scraping metrics from the "mgr/prometheus" module, no other Ceph alert condition will be detected, therefore creating a false sense of confidence.
You may start having a look at Prometheus and/or Alertmanager web UIs, or checking their logs.
Kind Regards, Ernesto
On Tue, Apr 15, 2025 at 7:28 PM Tim Holloway <timh@mousetech.com> wrote:
Although I've had this problem since at least Pacific, I'm still seeing it on Reef.
After much pain and suffering (covered elsewhere), I got my Prometheus services deployed as intended, Ceph health OK, green across the board.
However, over the weekend, the dreaded "CephMgrPrometheusModuleInactive" alert has returned to the Dashboard. "The mgr/prometheus module at dell02.mousetech.com:9283 is unreachable."
It's a blatant lie.
I still get "Ceph HEALTH_OK". All monitor status command show everything running. Checking ports on the host says it's listening.
More to the point, I can send my desktop browser to http://dell02.mousetech.com:9283 and get a page that will allow me to see the metrics. So everyone can see it but the Dashboard!
I did have some issues when the other prometheus host couldn't resolve the hostname, but I fixed that for all ceph hosts and it was green for days. Now the error is back. Restarting Prometheus didn't help.
How is the Dashboard hallucinating this???
Tim _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io