As suggested by someone, I tried `dump_historic_slow_ops`. There aren't many, and they're somewhat difficult to interpret: "description": "osd_op(client.250533532.0:56821 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3518464~8192] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299120+0000", "description": "osd_op(client.250533532.0:56822 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3559424~4096] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299132+0000", "description": "osd_op(client.250533532.0:56823 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3682304~4096] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299138+0000", "description": "osd_op(client.250533532.0:56824 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3772416~4096] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299148+0000", "description": "osd_op(client.250533532.0:56825 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3796992~8192] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299188+0000", "description": "osd_op(client.250533532.0:56826 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3862528~8192] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299198+0000", "description": "osd_op(client.250533532.0:56827 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3899392~12288] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299207+0000", "description": "osd_op(client.250533532.0:56828 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 3944448~16384] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299250+0000", "description": "osd_op(client.250533532.0:56829 13.16f 13:f6c9079e:::rbd_data.eed629ecc1f946.000000000000001c:head [stat,write 4018176~4096] snapc 0=[] ondisk+write+known_if_redirected e118835)", "initiated_at": "2023-04-26T07:00:58.299270+0000", There's a lot more information there ofc. I also tried to `dump_ops_in_flight` and there aren't many, usually 0-10 ops at a time, but the OSD latency remains high even when the ops count is low or zero. Any ideas? I would very much appreciate it if some could please point me to the documentation on interpreting the output of ops dump. /Z On Wed, 26 Apr 2023 at 20:22, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
I have a Ceph 16.2.12 cluster with uniform hardware, same drive make/model, etc. A particular OSD is showing higher latency than usual in `ceph osd perf`, usually mid to high tens of milliseconds while other OSDs show low single digits, although its drive's I/O stats don't look different from those of other drives. The workload is mainly random 4K reads and writes, the cluster is being used as Openstack VM storage.
Is there a way to trace, which particular PG, pool and disk image or object cause this OSD's excessive latency? Is there a way to tell Ceph to
I would appreciate any advice or pointers.
Best regards, Zakhar