Hi Joshua I don't think this is the case. The scrape interval for prom is 15 seconds. I also checked the debug logs for the prom module, and the metrics collection took less than 10 seconds most of the time. Chris. On Mon, Jul 28, 2025 at 5:54 PM Joshua Baergen <jbaergen@digitalocean.com> wrote:
Hi Chris,
Assuming that the scrape period for prom is set to 1 minute, you could simply be racing against the scrape. Usually it's not a good idea to create range vectors with the same time range as the scrape period. Given that you're using irate(), you could increase that to [2m] or higher and still get the effect that you want, since it'll always pick the two most recent points within the range on which to perform the rate calculation.
Josh
On Mon, Jul 28, 2025 at 5:40 AM Christopher James <christopher.jamesjr2000@gmail.com> wrote:
Hi,
We have encountered some issues regarding the RBD stats metrics.
We have some Grafana panels that show the per-pool rate of changes in RBD IOPS and Throughput.
We use these queries for IOPS round(sum(irate(ceph_rbd_write_ops[1m])) by (pool)) round(sum(irate(ceph_rbd_read_ops[1m])) by (pool))
and these queries for throughput round(sum(irate(ceph_rbd_write_bytes[1m])) by (pool)) round(sum(irate(ceph_rbd_read_bytes[1m])) by (pool))
Now, the problem we encounter here is that at some points in time, there are some odd outputs. In a way that the output has a lot of spikes, and at many points in time, there seems to be no change in the mentioned values, hence the irate()'s output would become zero, and the next data point would show that the
data
has somehow doubled from the last time.
I investigated the mgr/prometheus module to see if it is related to the cache mechanism, and it doesn't seem to be about that because the metrics collection method is done in less than 10 seconds most of the time and there is no "collecting data took more than ..." log.
I also investigated the raw time series data and saw that the two datapoints are exactly the same at the times that the irate() returns zero. I have put some screenshots of the panels in this read-only Google Doc:
https://docs.google.com/document/d/1Rf0dl4qAWnOtG80BsxVY9cjljQqliG2JHxgzVxDj...
It is somewhat odd, to be honest, that every image in every pool does not change in IOPS or throughput.
Is this the natural way the metrics are exposed? Should I not use
irate()?
Has anybody else encountered this issue as well?
Cheers, Chris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io