Hi all,
I am having trouble with our cluster getting consistent RBD latencies to our KVM virtual
machines connected via the KRBD driver. When measuring with tools like rbd perf image
iotop, we constantly see latency spike up from around 1-2ms to 100+ms. This seems to kill
our Windows VM SQL performance. I essentially have 2 questions:
1) Am I missing something with my configuration that should be applied to get consistent
low latency to the VM guests?
2) When measuring the disks, it seems that sequential IO results in higher latency vs
random IO. Is this correct or is there a way to tweak this using the KRBD driver?
Configuration:
3 x MON/MGR nodes
12 x OSD nodes (24 x HDD, 2 x NVMe for DB and WAL)
KVM clients attaching the RBD images via KRBD
1 pool w/ 16384 PGs
Ceph version 14.2.1
Ceph.conf:
[global]
mon host = 10.97.11.17,10.97.11.27,10.97.11.37
public network = 10.97.11.0/24
cluster network = 10.97.12.0/24
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd journal size = 30720
osd pool default size = 3
osd pool default min size = 2
osd pool default pg num = 4096
osd pool default pgp num = 4096
osd crush chooseleaf type = 1
[osd]
bluestore_default_buffered_write = false
bluestore_default_buffered_read = true
Thank you,
Show replies by date