On Thu, 10 Oct 2019, Xuehan Xu wrote:
My recollection is that rocksdb is always flushing, correct. There are conveniently only a handful of writers in rocksdb, the main ones being log files and sst files.
We could probably put an assertion in fsync() so ensure that the FileWriter buffer is empty and flushed...?
Thanks for your reply, sage:-) I will do that:-)
By the way, I've got another question here: It seems that BlueStore tries to provide some kind of atomic I/O mechanism in which data and metadata are either both modified or both untouched. To accomplish this, for modifications whose size is larger than prefer_defer_size, BlueStore will allocate new space for the modifications and release the old storage space. I think, in the long run, a initially contiguous stored file in bluestore could become scattered if there have been many random modifications to that file. Actually, this is what we are experiencing in our test clusters. The consequence is that after some period of random modification, the sequential read performance of that file is significantly degraded. Should we make this atomic I/O mechanism optional? It seems that most hard disk only make sure that a sector is never half-modified, for which, I think, the deferred I/O is enough. Am I right? Thanks:-)
I mean, in the scenario of RBD, since most real hard disk only guarantee that a sector is never half-modified, only providing atomic I/O guarantee for modifications whose are less than or equal to that of a disk sector, which is guaranteed by deferred io, should be enough. So, maybe, this atomic I/O guarantee for large size modifications should be made configurable.
The OSD needs to record both the data update *and* the metadata associated with it (pg log entry) atomically, so atomic sector updates aren't sufficient. You might try looking at the bluestore_prefer_deferred_size, which will make writes take the deferred IO path. This gets increasingly inefficient the larger the value is, though! If we really find that fragmentation is a problem over the long term, we should make the deep scrub process rewrite the data it has read if/when it is too fragmented. sage