Hi Roberto, thank you very much, this is the right parameter to toggle! I have change it from "false" to "true" and now we have graphs working again. Have a good day Francesco Il 25/02/26 14:20, Roberto Valverde Cameselle ha scritto:

Hi Francesco, Can you check if you have this feature set? mgr               advanced  mgr/prometheus/exclude_perf_counters Make sure that is set to true so then performance metrics are gathered from the ceph-exporters and not from the manager. If your cluster is small, you can also not add ceph-exporters and set the above setting to false so the manager will be also exposing the performance metrics. Cheers

*-----------------------------------------------------------------* *Roberto VALVERDE CAMESELLE* IT Storage And Data Management CERN, European Organization for Nuclear Research Esplanade des Particules 1, Geneve (Switzerland) ------------------------------------------------------------------------ *From:* Francesco Usseglio Gaudi via ceph-users <ceph-users@ceph.io> *Sent:* Tuesday, February 24, 2026 6:17 PM *To:* ceph-users@ceph.io <ceph-users@ceph.io> *Subject:* [ceph-users] metrics duplicate in prometheus after migrate and upgrade to cephadm Hi all,

we have a 10 nodes cluster used for S3 with radosgw.

Initially the cluster has been installed with ceph-ansible and was at version 18.

Some months ago an external consultant converted to cephadm and upgrade to 19.2.3

Almost everything is ok but we have some issues with dashboard and grafana graphs.

In Dashboard → Cluster → OSDs → Overall Performance we can't see "OSD Read Latencies" and "OSD Write Latencies" graphs, they have red triangle icons with this error popup:

"execution: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{ceph_daemon=\"osd.0\", cluster=\"6e06959e-3ef3-4017-a467-b1d482bc7269\", instance=\"ceph_cluster\", job=\"ceph\"}, {ceph_daemon=\"osd.0\", cluster=\"6e06959e-3ef3-4017-a467-b1d482bc7269\", instance=\"node1\", job=\"ceph-exporter\"}];many-to-many matching not allowed: matching labels must be unique on one side"

After some my investigations I found out there are some duplicate metrics in prometheus db, one coming from ceph (I think mgr module?) and one from ceph-exporter, for example  this metric:

ceph_osd_op_r_latency_sum

be there with two distinct values with different job labels (values are exactly the same)

Every node has a running ceph-exporter container and there are 3 nodes with mgr services.

I have tried to modify grafana dashboard queries to use only one of two metrics specifying the job label but they are readonly (I can make a copy and modifying it that works, but i don't know how to use this new grafana dashboard in ceph dashboard).

I have read from ceph documentation and I understand ceph-exporter is no longer required but on another ceph cluster recently set up with cephadm version 19.2.3 from the very start, ceph-exporter container are presents on every nodes, so I'm not sure if I should keep ceph-exporter containers running

Which is the right way of extract these metrics from ceph?

Thank you all

Francesco Usseglio _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io