Hi all!

I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard).

I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error

When I access the node shown as down, it reports

503 Service Unavailable

No cached data available yet

Traceback (most recent call last):
  File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond
    self._do_respond(path_info)
  File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond
    response.body = self.handler()
  File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics
    return self._metrics(_global_instance)
  File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics
    raise cherrypy.HTTPError(503, 'No cached data available yet')
cherrypy._cperror.HTTPError: (503, 'No cached data available yet')

I checked the mgr prometheus address and port

It seems to be ok.

When I check the master manager node for the port, I found