Prometheus metrics seem to be broken, too: ceph_osd_op_r_latency_sum ceph_osd_op_w_latency_sum Both of them for example are not reported by the ceph mgr metrics exporter: curl http://192.168.XXX.XXX:9283/metrics |grep ceph_osd_ I get some merics like "ceph_osd_commit_latency_ms" or "ceph_osd_apply_latency_ms" but none of the "op_r / op_w" metrics. Do the lable names have switched and my grafana dashboard is outdated? Or are they missing at the exporter level? Thanks ________________________________ Von: Kilian Ries Gesendet: Montag, 25. November 2024 13:37:16 An: ceph-users@ceph.io Betreff: AW: Ceph OSD perf metrics missing Any ideas? Still facing the problem ... ________________________________ Von: Kilian Ries Gesendet: Mittwoch, 23. Oktober 2024 13:59:06 An: ceph-users@ceph.io Betreff: Ceph OSD perf metrics missing Hi, i'm running a Ceph v18.2.4 cluster. I'm trying to build some latency monitoring with the ceph daemon osd.4 perf dump cli command. On most of the OSDs i get all the metrics i need. On some OSDs i only get zero values: osd.op_latency.avgcount: 0 I already tried restarting the OSD process which didn't help. I also tried to reset the metrics via ceph daemon osd.4 perf reset all but that didn't help either. How can that be that some OSDs don't show values here? How can i fix that? Thanks Regards, Kilian

Hello Kilian, I am not entirely sure this is the same issue, but I know Prometheus doesn’t export some performance metrics by default anymore in Reef. This can be worked around by setting: ceph config set mgr mgr/prometheus/exclude_perf_counters false A proper fix would be to switch to using a per-host ceph-exporter. Best regards, Pierre Riteau On Mon, 25 Nov 2024 at 13:45, Kilian Ries <mail@kilian-ries.de> wrote:

Prometheus metrics seem to be broken, too:

ceph_osd_op_r_latency_sum

ceph_osd_op_w_latency_sum

Both of them for example are not reported by the ceph mgr metrics exporter:

curl http://192.168.XXX.XXX:9283/metrics |grep ceph_osd_

I get some merics like "ceph_osd_commit_latency_ms" or "ceph_osd_apply_latency_ms" but none of the "op_r / op_w" metrics. Do the lable names have switched and my grafana dashboard is outdated? Or are they missing at the exporter level?

Thanks

________________________________ Von: Kilian Ries Gesendet: Montag, 25. November 2024 13:37:16 An: ceph-users@ceph.io Betreff: AW: Ceph OSD perf metrics missing

Any ideas? Still facing the problem ...

________________________________ Von: Kilian Ries Gesendet: Mittwoch, 23. Oktober 2024 13:59:06 An: ceph-users@ceph.io Betreff: Ceph OSD perf metrics missing

Hi,

i'm running a Ceph v18.2.4 cluster. I'm trying to build some latency monitoring with the

ceph daemon osd.4 perf dump

cli command. On most of the OSDs i get all the metrics i need. On some OSDs i only get zero values:

osd.op_latency.avgcount: 0

I already tried restarting the OSD process which didn't help. I also tried to reset the metrics via

ceph daemon osd.4 perf reset all

but that didn't help either. How can that be that some OSDs don't show values here? How can i fix that?

Thanks

Regards,

Kilian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io