[Ceph-users] Ceph RBD High Latency

Hi all, I am having trouble with our cluster getting consistent RBD latencies to our KVM virtual machines connected via the KRBD driver. When measuring with tools like rbd perf image iotop, we constantly see latency spike up from around 1-2ms to 100+ms. This seems to kill our Windows VM SQL performance. I essentially have 2 questions: 1) Am I missing something with my configuration that should be applied to get consistent low latency to the VM guests? 2) When measuring the disks, it seems that sequential IO results in higher latency vs random IO. Is this correct or is there a way to tweak this using the KRBD driver? Configuration: 3 x MON/MGR nodes 12 x OSD nodes (24 x HDD, 2 x NVMe for DB and WAL) KVM clients attaching the RBD images via KRBD 1 pool w/ 16384 PGs Ceph version 14.2.1 Ceph.conf: [global] mon host = 10.97.11.17,10.97.11.27,10.97.11.37 public network = 10.97.11.0/24 cluster network = 10.97.12.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx osd journal size = 30720 osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 4096 osd pool default pgp num = 4096 osd crush chooseleaf type = 1 [osd] bluestore_default_buffered_write = false bluestore_default_buffered_read = true Thank you,