Hi,
Our cluster (14.2.6) has sporadic slow ops warnings since upgrading from Jewel 1 month
ago. Today I checked the OSD log files and found out a lot of entries like:
ceph-osd.5.log:2020-03-04 10:33:31.592 7f18ca41f700 0 bluestore(/var/lib/ceph/osd/ceph-5)
log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.16871s, txc =
0x55e33ae41b80
ceph-osd.5.log:2020-03-04 10:33:31.592 7f18ca41f700 0 bluestore(/var/lib/ceph/osd/ceph-5)
log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.15158s, txc =
0x55e3639b3340
ceph-osd.5.log:2020-03-04 10:33:31.592 7f18ca41f700 0 bluestore(/var/lib/ceph/osd/ceph-5)
log_latency_fn slow operation observed for _txc_committed_kv, latency = 6.77361s, txc =
0x55e3379cc840
ceph-osd.5.log:2020-03-04 10:33:52.666 7f18ca41f700 0 bluestore(/var/lib/ceph/osd/ceph-5)
log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.42519s, txc =
0x55e33722d600
or
/var/log/kolla/ceph/ceph-osd.7.log:2020-03-04 00:41:31.110 7f3dc0bc8700 0
bluestore(/var/lib/ceph/osd/ceph-7) log_latency slow operation observed for
submit_transact, latency = 8.1279s
/var/log/kolla/ceph/ceph-osd.7.log:2020-03-04 00:41:31.110 7f3dd1bea700 0
bluestore(/var/lib/ceph/osd/ceph-7) log_latency slow operation observed for kv_final,
latency = 7.88786s
/var/log/kolla/ceph/ceph-osd.7.log:2020-03-04 02:21:35.180 7f3dd1bea700 0
bluestore(/var/lib/ceph/osd/ceph-7) log_latency slow operation observed for kv_final,
latency = 6.06171s
/var/log/kolla/ceph/ceph-osd.7.log:2020-03-04 05:31:30.298 7f3dc1bca700 0
bluestore(/var/lib/ceph/osd/ceph-7) log_latency slow operation observed for
submit_transact, latency = 5.34228s
The cluster setup is: SATA SSD (as DB) + SATA HDD 1:3.
Any suggest how to debug this problem? Thank you!
br,
Xu Yun