On Sat, Mar 25, 2023 at 11:09:58AM +0700, Konstantin Shalygin wrote:
Hi Matthias,
Prometheus exporter already have all this metrics, you can setup Grafana panels as you
want
Also, the apply latency in a metric for a pre-bluestore, i.e. filestore
For Bluestore apply latency is the same as commit latency, you can check this via `ceph
osd perf` command
Thanks Konstantin,
do I guess right that the metrics shown in your screenshot correspond to
values
"bluestore.txc_commit_lat.description": "Average commit latency",
"bluestore.txc_throttle_lat.description": "Average submit throttle
latency",
"bluestore.txc_submit_lat.description": "Average submit latency",
"bluestore.read_lat.description": "Average read latency",
from "ceph daemon osd.X perf dump"?
And "ceph osd perf" output would correspond to
"bluestore.txc_commit_lat.description": "Average commit latency",
or
"filestore.apply_latency.description": "Apply latency",
"filestore.journal_latency.description": "Average journal queue
completing latency",
depending on OSD format?
It looks like "read_lat" is Bluestore only, and there is no comparable
value for Filestore.
There are other, format-agnostic OSD latency values:
"osd.op_r_latency.description": "Latency of read operation (including
queue time)",
"osd.op_w_latency.description": "Latency of write operation (including
queue time)",
"osd.op_rw_latency.description": "Latency of read-modify-write operation
(including queue time)",
More guesswork:
- is osd.op_X_latency about client->OSD command timing?
- are bluestore/filestore values about OSD->storage op timing?
Please bear with me :-) I just try to get some rough understanding what
the numbers to be collected and graphed actually mean and how they are
related to each other.
Regards
Matthias
> > On 25 Mar 2023, at 00:02, Matthias Ferdinand <mf+ml.ceph(a)mfedv.net>
wrote:
> >
> > Hi,
> >
> > I would like to understand how the per-OSD data from "ceph osd perf"
> > (i.e. apply_latency, commit_latency) is generated. So far I couldn't
> > find documentation on this. "ceph osd perf" output is nice for a
quick
> > glimpse, but is not very well suited for graphing. Output values are
> > from the most recent 5s-averages apparently.
> >
> > With "ceph daemon osd.X perf dump" OTOH, you get quite a lot of
latency
> > metrics, while it is just not obvious to me how they aggregate into
> > apply_latency and commit_latency. Or some comparably easy read latency
> > metric (something that is missing completely in "ceph osd perf").
> >
> > Can somebody shed some light on this?
> >
> >
> > Regards
> > Matthias
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io