You may find this interesting. I'm running Pacific from the Red Hat repo and Prometheus was given its own discrete container image, not the generic Ceph one. Rather than build custom Prometheus, Red Hat used the Prometheus project's own containers. In fact, it has 3: one for Prometheus, one for Prometheus node-exporter and one for Prometheus alertmanager. Also Grafana is using its own image as well. Nothing on the host in question listens on Port 8765. Prometheus is listening on ports in the 9000 range. For whatever it's worth. The name of the API you're interested in, on the other hand, sounds familiar, but I'm too senile to recall from where. Tim On Tue, 2024-09-03 at 11:48 -0400, Tim Holloway wrote:
Yeah. Although taming the Prometheus logs is on my list, I'm still fuzzy on its details.
For your purposes, Docker and Podman can be considered as equivalent. I also run under Podman, incidentally.
If the port isn't open inside the container, then blame Prometheus. I'd consider bumping its logging levels - the exact opposite of what I'm looking to do!
Tim
On Tue, 2024-09-03 at 15:34 +0100, Matthew Vernon wrote:
Hi,
On 03/09/2024 14:27, Tim Holloway wrote:
FWIW, I'm using podman not docker.
The netstat command is not available in the stock Ceph containers, but the "ss" command is, so use that to see if there is in fact a process listening on that port.
I have done this, and there's nothing listening on port 8765 per ss -l | grep 8765
If not, then the process that listens apparently didn't come all the way up.
Take a quick peek around the local /var/log directory just to see if there's anything logged internally (probably not). Maybe do a "dmesg" just for laughs.
Nothing obvious - and logs are available outside the container as well - hence the paste earlier in this thread, https://phabricator.wikimedia.org/P68521
Since this is (so I read it), the Prometheus container, look at the /var/lib/ceph/<fsid>/prometheus.<hostname> directory, which contains the Prometheus config file (under "./etc) and see if there are irregularities there.
It's not, it's the mgr container (I've enabled the prometheus mgr module, which makes an endpoint available from whence metrics can be scraped, rather than the prometheus container which runs the service that does the scraping).
Thanks,
Matthew _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io