Thanks, Eugen! It's a bunch of entries like this https://pastebin.com/TGPu6PAT - I'm not really sure what to make of them. I checked adjacent OSDs and they have similar ops, but aren't showing excessive latency. /Z On Thu, 27 Apr 2023 at 10:42, Eugen Block <eblock@nde.ag> wrote:
Hi,
I would monitor the historic_ops_by_duration for a while and see if any specific operation takes unusually long.
# this is within the container [ceph: root@storage01 /]# ceph daemon osd.0 dump_historic_ops_by_duration | head { "size": 20, "duration": 600, "ops": [ { "description": "osd_repop(client.9384193.0:2056545 12.6 e2233/2221 12:6192870f:::obj_delete_at_hint.0000000053:head v 2233'696390, mlcod=2233'696388)", "initiated_at": "2023-04-27T07:37:35.046036+0000", "age": 54.805016199999997, "duration": 0.58198468699999995, ...
The output contains the PG (so you know which pool is involved) and the duration of the operation, not sure if that helps though.
Zitat von Zakhar Kirpichenko <zakhar@gmail.com>:
As suggested by someone, I tried `dump_historic_slow_ops`. There aren't many, and they're somewhat difficult to interpret:
"description": "osd_op(client.250533532.0:56821 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3518464~8192] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299120+0000", "description": "osd_op(client.250533532.0:56822 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3559424~4096] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299132+0000", "description": "osd_op(client.250533532.0:56823 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3682304~4096] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299138+0000", "description": "osd_op(client.250533532.0:56824 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3772416~4096] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299148+0000", "description": "osd_op(client.250533532.0:56825 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3796992~8192] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299188+0000", "description": "osd_op(client.250533532.0:56826 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3862528~8192] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299198+0000", "description": "osd_op(client.250533532.0:56827 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3899392~12288] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299207+0000", "description": "osd_op(client.250533532.0:56828 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3944448~16384] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299250+0000", "description": "osd_op(client.250533532.0:56829 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 4018176~4096] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299270+0000",
There's a lot more information there ofc. I also tried to `dump_ops_in_flight` and there aren't many, usually 0-10 ops at a time, but the OSD latency remains high even when the ops count is low or zero. Any ideas?
I would very much appreciate it if some could please point me to the documentation on interpreting the output of ops dump.
/Z
On Wed, 26 Apr 2023 at 20:22, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
I have a Ceph 16.2.12 cluster with uniform hardware, same drive make/model, etc. A particular OSD is showing higher latency than usual in `ceph osd perf`, usually mid to high tens of milliseconds while other OSDs show low single digits, although its drive's I/O stats don't look different from those of other drives. The workload is mainly random 4K reads and writes, the cluster is being used as Openstack VM storage.
Is there a way to trace, which particular PG, pool and disk image or object cause this OSD's excessive latency? Is there a way to tell Ceph to
I would appreciate any advice or pointers.
Best regards, Zakhar
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io