[ceph-users] RBD huge diff between random vs non-random IOPs - all flash

Hi all, I'm trying to troubleshoot an interesting problem with RBD performance for VMs. Tests were done using fio both outside and inside the VMs shows that random read/write is 20-30% slower than bulk read/write at QD=1. However, at QD=16/32/64, random read/write is sometimes 3X faster than bulk read/write. Inside the VMs, tests were done with -direct=1 -sync=1 using libaio. Outside VMs, test were done with -direct=1 -sync=1 with both librbd and libaio. The gap between random and bulk I/O narrows with increasing QD to 128. However, there's always a 20-30% difference with random I/O being faster. Read and write tests show similar results both inside and outside VMs. Typically, the random I/O performance would be less (or much less) than bulk. Any idea as to what I should be looking at? Thanks. Tri Hoang Inside VM ======== At QD=8, randread is around 2.8X read @ 64k --------------------------------------------------------------- tri@ansible:~$ fio -name=read -ioengine=libaio -iodepth=8 -direct=1 -sync=1 -rw=randread -bs=64k -size=4G -runtime=120 --filename=test.fio read: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=8 fio-3.12 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=762MiB/s][r=12.2k IOPS][eta 00m:00s] read: (groupid=0, jobs=1): err= 0: pid=1551: Wed Sep 30 08:19:56 2020 read: IOPS=11.9k, BW=743MiB/s (779MB/s)(4096MiB/5515msec) tri@ansible:~$ fio -name=read -ioengine=libaio -iodepth=8 -direct=1 -sync=1 -rw=read -bs=64k -size=4G -runtime=120 --filename=test.fio read: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=8 fio-3.12 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=268MiB/s][r=4289 IOPS][eta 00m:00s] read: (groupid=0, jobs=1): err= 0: pid=1554: Wed Sep 30 08:22:03 2020 read: IOPS=4374, BW=273MiB/s (287MB/s)(4096MiB/14981msec) At QD=128, randread is around 1.4X read --------------------------------------------------------- tri@ansible:~$ fio -name=read -ioengine=libaio -iodepth=128 -direct=1 -sync=1 -rw=randread -bs=64k -size=4G -runtime=120 --filename=test.fio read: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128 fio-3.12 Starting 1 process Jobs: 1 (f=1) read: (groupid=0, jobs=1): err= 0: pid=1548: Wed Sep 30 08:18:59 2020 read: IOPS=23.1k, BW=1441MiB/s (1511MB/s)(4096MiB/2843msec) tri@ansible:~$ fio -name=read -ioengine=libaio -iodepth=128 -direct=1 -sync=1 -rw=read -bs=64k -size=4G -runtime=120 --filename=test.fio read: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128 fio-3.12 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=974MiB/s][r=15.6k IOPS][eta 00m:00s] read: (groupid=0, jobs=1): err= 0: pid=1545: Wed Sep 30 08:17:38 2020 read: IOPS=15.9k, BW=997MiB/s (1045MB/s)(4096MiB/4110msec)