2 Jun
2020
2 Jun
'20
4:40 a.m.
Hi Yiming,
Are you changing the overall data set size when you change the image
size? IE in your 40GB image test, is your data set 40x larger than in
your 1GB image test? That would have various effects, including
changing the number of onodes in the cache and the potential for cache
misses hitting rocksdb and eventually the disk. Having said that, with
the default 4GB memory target I wouldn't expect you to have cache misses
with typical RBD workloads even with a 40GB dataset on a single OSD
unless you've tweaked the object size to be smaller or caused additional
metadata per object in some way (EC, etc).
Theoretically you might be able to use lttng or jaeger tracepoints to
track latency, or possible look at the perf counters. Otherwise you
might also be able to see something through wallclock profiling. I
would probably look carefully at things happening in the kv sync thread
since this is a random write workload and that's where I'd expect to see
blocking behavior that could cause latency spikes like this.
Mark
On 6/1/20 1:50 PM, Yiming Zhang wrote:
Hi All,
I have noticed that different RBD image size can shape the bluestore
latency differently. Is there baseline or guidance for choosing the
image size?
Left: RBD image size is 1GB
middle: RBD image size is 40GB
Right: RBD image size is 1GB, RocksDB write buffer 10X default
4K randwrite on SSD with FIO. SSD is preconditioned and image is
prefilled(20mins).
Red dot is L1 compaction and green dot is L0 compaction.
Let’s focus on the left graph. The smaller spikes are caused by
compactions. The higher spikes seems to be caused by the BlueStore
itself.
I suspect this could be related to RBD image size in someway.
Does anyone know what could the cause of the higher spikes? And how to
debug it?
Also, what is the proper RBD image size for my test?
Please advice.
Thanks,
Yiming
_______________________________________________
Dev mailing list -- dev(a)ceph.io
To unsubscribe send an email to dev-leave(a)ceph.io