Hi, What do people use to store per-RBD-image I/O stats? To retrospectively see which VM was keeping a Ceph cluster busy in terms of I/O, I'd like to track `rbd perf image iostat` and `rbd perf image iotop`-like metrics. I've looked at the LibreNMS plugin [1], Prometheus metrics in the manager [2], and Ceph ingestion in Elasticsearch [3]. None of them seem to export per-RBD metrics. I can't find much information about this use case on the mailing list either, though it's hard for me to imagine I'm the only one with this want. How do other users handle this? William David Edwards [1]: https://github.com/librenms/librenms-agent/blob/master/agent-local/ceph [2]: https://docs.ceph.com/en/latest/mgr/prometheus/ [3]: https://www.elastic.co/docs/reference/integrations/ceph
That can't be true, I have a grafana dashboard somewhere that shows this. I think you need to enable plugin in mgr or so.
What do people use to store per-RBD-image I/O stats?
To retrospectively see which VM was keeping a Ceph cluster busy in terms of I/O, I'd like to track `rbd perf image iostat` and `rbd perf image iotop`-like metrics.
I've looked at the LibreNMS plugin [1], Prometheus metrics in the manager [2], and Ceph ingestion in Elasticsearch [3].
None of them seem to export per-RBD metrics. I can't find much information about this use case on the mailing list either, though it's hard for me to imagine I'm the only one with this want.
How do other users handle this?
Correct, its collection is disabled per default for performance reasons [0]. To enable per-pool statistics, you can either provide a list of pools, or use a wildcard as described in [1]: ceph config set mgr mgr/prometheus/rbd_stats_pools "*" And you might need to enable the iostat mgr module: ceph mgr module enable iostat But I assume that the module is already enabled since you are able to see the stats with rbd perf image iostat. Grafana then should display the stats. [0] https://docs.ceph.com/en/latest/cephadm/services/monitoring/#setting-up-rbd-... [1] https://docs.ceph.com/en/latest/mgr/prometheus/#prometheus-rbd-io-statistics Zitat von Marc <Marc@f1-outsourcing.eu>:
That can't be true, I have a grafana dashboard somewhere that shows this. I think you need to enable plugin in mgr or so.
What do people use to store per-RBD-image I/O stats?
To retrospectively see which VM was keeping a Ceph cluster busy in terms of I/O, I'd like to track `rbd perf image iostat` and `rbd perf image iotop`-like metrics.
I've looked at the LibreNMS plugin [1], Prometheus metrics in the manager [2], and Ceph ingestion in Elasticsearch [3].
None of them seem to export per-RBD metrics. I can't find much information about this use case on the mailing list either, though it's hard for me to imagine I'm the only one with this want.
How do other users handle this?
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Is there more information on "performance reasons" [0]? Elsewhere in the documentation [1] it even states:
Monitoring of RBD images is disabled by default, as it can significantly impact performance.
We want to use this in production to hopefully help with debugging a rare nightly latency issue. Obviously, we don't want to introduce new performance issues. Does anyone have experience with enabling RBD per-image I/O statistics [2] in production? Did it cause you any performance (or other) issues? Mika [0] https://docs.ceph.com/en/latest/cephadm/services/monitoring/#setting-up-rbd-... [1] https://docs.ceph.com/en/latest/mgr/dashboard/#enabling-rbd-image-monitoring [2] https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics
If you have thousands of RBD volumes, the mgr might get bogged down. Turn it on for your pool of interest while you watch and see what happens. Re a nightly latency issue, does it correspond with a spike in read/write throughput? When I hear of a nightly perf issue, the first thing I have to ask is if you have [m]locate disabled on your clients. Cron jobs to update the DB can have this effect, especially if the guest OS doesn't add a random slew to the invocation - and of course if the backing pool is on spinners. Another idea is if you have deep scrubs limited to certain hours., or very verbose logs on the clients that are being rotated and compressed at the same time.
On Sep 16, 2025, at 4:48 AM, Mika <mscheer@cyberfusion.io> wrote:
Is there more information on "performance reasons" [0]?
Elsewhere in the documentation [1] it even states:
Monitoring of RBD images is disabled by default, as it can significantly impact performance.
We want to use this in production to hopefully help with debugging a rare nightly latency issue. Obviously, we don't want to introduce new performance issues.
Does anyone have experience with enabling RBD per-image I/O statistics [2] in production? Did it cause you any performance (or other) issues?
Mika
[0]
https://docs.ceph.com/en/latest/cephadm/services/monitoring/#setting-up-rbd-...
[1]
https://docs.ceph.com/en/latest/mgr/dashboard/#enabling-rbd-image-monitoring
[2]
https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Anthony D'Atri
-
Eugen Block
-
Marc
-
Mika
-
William David Edwards