Wrong hostnames in "ceph mgr services" (Octopus)
Hi, After upgrading from 15.2.8. to 15.2.13 with cephadm on CentOS 8 (containerised installation done by cephadm), Grafana no longer shows new data. Additionally, when accessing the Dashboard-URL on a host currently not hosting the dashboard, I am redirected to a wrong hostname (as shown in ceph mgr services). I assume that this is caused by the same reason which leads to this output of `ceph mgr services`: { "dashboard": "https://ceph-<cluster-id>-mgr.iceph-11.tsmsqs:8443/", "prometheus": "http://ceph-<cluster-id>-mgr.iceph-11.tsmsqs:9283/" } The correct hostname is iceph-11 (without the tsmsqs part), FQDN is iceph-11.servernet. The hosts use DNS, the names (iceph-11 and iceph-11.servernet) are resolvable both from the hosts as well as from within the Podman containers. I have determined that podman by default sets the container name as a hostname alias (visible with `hostname -a` within the container), which somehow leads to Ceph mgr picking it up as the primary name? My workaround is to modify /var/lib/ceph/<cluster-id>/mgr.<hostname>.<random-6-char-string>/unit.run, adding --no-hosts as an additional argument to the "podman run" command. I could probably use a system-wide containers.conf as well. With this workaround and after restarting the Ceph mgr container (via systemctl) and then restarting Prometheus and Grafana (with ceph orch redeploy), I once again get data in Grafana and the correct redirect for the dashboard. `ceph mgr services` also shows expected and correct values. I am wondering if this kind of issue is known or whether there is something wrong with my setup. I expected Ceph mgr to use the primary hostname and not some seemingly random hostname alias. Maybe this issue can also be discussed in a troubleshooting section of the monitoring stack documentation. Cheers Sebastian
participants (1)
-
Sebastian Knust