I have noticed that different RBD image size can shape the bluestore latency differently. Is there baseline or guidance for choosing the image size?

Left: RBD image size is 1GB 

middle: RBD image size is 40GB

Right: RBD image size is 1GB, RocksDB write buffer 10X default

4K randwrite on SSD with FIO. SSD is preconditioned and image is prefilled(20mins). 

Red dot is L1 compaction and green dot is L0 compaction.

Let’s focus on the left graph. The smaller spikes are caused by compactions. The higher spikes seems to be caused by the BlueStore itself. 

I suspect this could be related to RBD image size in someway.

Does anyone know what could the cause of the higher spikes? And how to debug it? 

Also, what is the proper RBD image size for my test?

Please advice. 

Thanks,

Yiming