Thank you, Redouane! Some background. I migrated to Ceph amidst a Perfect Storm. The Ceph docs, as I've often complained, were/are a horrible mish-mash of deprecated instructions and more modern information. So, among other things, I ended up with a mess of resources, some legacy-based, some managed, and some OSDs that managed to get defined as both. On top of that, this was Ceph Octopus, which had the flaw that scheduled changes to the system would hang until the system was in OK status, but the way to make the system OK was to have those scheduled changes actually executed. I managed to clear that part out by migrating to Pacific. I originally settled on Octopus because I wrongly had thought that no newer release was supported standard under CentOS 7. Over time I thought I'd managed to get the system clean, via a combination of documented processes and occasional brute force, but it looks like something may not have been configured/activated, since as I said, I see no traffic targeting port 8765 not anyone listening there. So if you could guide me on fixing that, I'd be grateful. Tim On Fri, 2024-09-06 at 09:08 +0200, Redouane Kachach wrote:
Hi Matthew,
That makes sense. The ipv6 BUG can lead to the issue you described. In the current implementation whenever a mgr failover takes place, prometheus configuration (when using the monitoring stack deployed by Ceph) is updated automatically to point to the new active mgr. Unfortunately it's not easy to have active services running in the standby mgr. At most, we can do some redirection as we do in the dashboard. So far we haven't had the need to do that. Next releases will come with the new mgmt-gateway service introduced in [1] and [2] which will make it easy to have a single entry point to the cluster handling HA transparently in the backend. This is still WIP but you can play with it if you want using the latest code from main. Support for OIDC based on oauth2-proxy is also being introduced as part of this effort by [3].
@ Timo Holloway, as I said the support [4] for service discovery has been there for a while (I'd say 2 years aprox) unless you are using an old Ceph version (where the prometheus config was static) you should see traffic in the port 8765.
[1] https://github.com/ceph/ceph/pull/57535 [2] https://github.com/ceph/ceph/pull/58402 [3] https://github.com/ceph/ceph/pull/58460 [4] https://github.com/ceph/ceph/pull/46400
On Thu, Sep 5, 2024 at 7:00 PM Tim Holloway <timh@mousetech.com> wrote:
Now you've got me worried. As I said, there is absolutely no traffic using port 8765 on my LAN.
Am I missing a service? Since my distro is based on stock Prometheus, I'd have to assume that the port 8765 server would be part of the Ceph generic container image and isn't being switched on for some reason.
Tim