OK. Found it. The primary prometheus node had a bad /etc/hosts. Most of my ceph nodes are on their own sub-domain, but a few have legacy domain names and the uncontacted node is one of them. Since I have wildcard resolution on DNS, Ceph was polling the wrong machine instead of failing to resolve outright and it wasn't obvious because the /etc/hosts on the other ceph nodes was set properly. The odd thing was, I thought I'd addressed that a while back, and as I said things HAD been working. But when I started tracking into the inner workings, I found that the entry had apparently reverted despite the fact that I've done no further maintenance on them. I'll double-check my master provisioner and if the problem comes back at least I'll know to look more carefully. Pity the dashboard error doesn't include the failing IP address along with the URL. Anyway, thanks all for the help! Tim On 4/21/25 09:46, Tim Holloway wrote:
Thanks, but all I'm getting is the following every 10 minutes from the prometheus nodes:
Apr 21 09:29:32 dell02.mousetech.com podman[997331]: 2025-04-21 09:29:32.252358201 -0400 EDT m=+0.039016913 container exec 2149e16fa2ce8769bf3be9e6e25eec61b8e027b0e8699f1cb7d5f113fc4aac66 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-278fcd86-0861-11ee-a7df-9c5c8e86cf8f-node-exporter-dell02, maintainer=The Prometheus Authors <prometheus-developers@googlegroups.com>) Apr 21 09:29:32 dell02.mousetech.com podman[997331]: 2025-04-21 09:29:32.259543374 -0400 EDT m=+0.046202087 container exec_died 2149e16fa2ce8769bf3be9e6e25eec61b8e027b0e8699f1cb7d5f113fc4aac66 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-278fcd86-0861-11ee-a7df-9c5c8e86cf8f-node-exporter-dell02, maintainer=The Prometheus Authors <prometheus-developers@googlegroups.com>) Apr 21 09:29:32 dell02.mousetech.com podman[997518]: 2025-04-21 09:29:32.947995363 -0400 EDT m=+0.036761633 container exec 71c75380a0b63fa8a31fa296c733d8385309b44325813450fcd30670b249157c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-278fcd86-0861-11ee-a7df-9c5c8e86cf8f-prometheus-dell02, maintainer=The Prometheus Authors <prometheus-developers@googlegroups.com>) Apr 21 09:29:32 dell02.mousetech.com podman[997518]: 2025-04-21 09:29:32.979516297 -0400 EDT m=+0.068282565 container exec_died 71c75380a0b63fa8a31fa296c733d8385309b44325813450fcd30670b249157c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-278fcd86-0861-11ee-a7df-9c5c8e86cf8f-prometheus-dell02, maintainer=The Prometheus Authors <prometheus-developers@googlegroups.com>)
It looks like a process is failing inside the prometheus container, but the container itself remains in operation.
On another note, I was all green until 2 days ago when a node went into overload. The node has recovered, but the prometheus alert is back and has been for 2 days.
Tim
On 4/21/25 07:25, Ernesto Puerta wrote:
You could check Alertmanager container logs <https://docs.ceph.com/en/quincy/cephadm/operations/#example-of-logging-to-journald>
.
Kind Regards, Ernesto
On Wed, Apr 16, 2025 at 4:54 PM Tim Holloway <timh@mousetech.com> wrote:
I'm thinking more some sort of latency error.
I have 2 prometheus daemons running at the moment. The hosts files on all my ceph servers contain both hostname and FQDN.
This morning the alert was gone. I don't know where I might find a log of when it comes and goes, but all was clean, then it wasn't, now it's clean again and I haven't been playing with any sort of configurations or bouncing hosts or services. It's just appearing and disappearing.
Tim
On 4/16/25 09:34, Ankush Behl wrote:
Just to add upon what Ernesto mentioned. Your prometheus container might not be able to reachout to ceph scrape job as the it could be using FQDN or Hostname. Try updating /etc/hosts with ip and hostname of the ceph scrape job(you can find it on prometheus UI -> status -> targets) and restarting the prometheus after that might help resolve the issue.
On Wed, Apr 16, 2025 at 2:10 PM Ernesto Puerta <epuertat@redhat.com> wrote:
Don't shoot the messenger. Dashboard is just displaying the alert that Prometheus/AlertManager is reporting. The alert definition is here <
https://github.com/ceph/ceph/blob/3993779cde9d10512f4a26f87487d11103ac1bd0/m...
. As you may see, it's based on the status of the Prometheus "ceph" scrape job. This alert is vital, because if the "ceph" job is not scraping metrics from the "mgr/prometheus" module, no other Ceph alert condition will be detected, therefore creating a false sense of confidence.
You may start having a look at Prometheus and/or Alertmanager web UIs, or checking their logs.
Kind Regards, Ernesto
On Tue, Apr 15, 2025 at 7:28 PM Tim Holloway <timh@mousetech.com> wrote:
Although I've had this problem since at least Pacific, I'm still seeing it on Reef.
After much pain and suffering (covered elsewhere), I got my Prometheus services deployed as intended, Ceph health OK, green across the board.
However, over the weekend, the dreaded "CephMgrPrometheusModuleInactive" alert has returned to the Dashboard. "The mgr/prometheus module at dell02.mousetech.com:9283 is unreachable."
It's a blatant lie.
I still get "Ceph HEALTH_OK". All monitor status command show everything running. Checking ports on the host says it's listening.
More to the point, I can send my desktop browser to http://dell02.mousetech.com:9283 and get a page that will allow me to see the metrics. So everyone can see it but the Dashboard!
I did have some issues when the other prometheus host couldn't resolve the hostname, but I fixed that for all ceph hosts and it was green for days. Now the error is back. Restarting Prometheus didn't help.
How is the Dashboard hallucinating this???
Tim _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io