On 24/04/2025 13:57, 段世博 wrote:
We are currently using the Ceph Octopus version and have some questions regarding a specific commit in librbd ( https://github.com/ceph/ceph/commit/081d28ae7ca46fd1f40034cc558def77a95a9294 ).
Could you please clarify why Ceph implements ordering for overlapping IO? Our understanding is that overlapping IO in block storage can typically be parallelized. We are curious about which components of Ceph depend on this feature. Additionally, we noticed that this feature was removed in the subsequent Pacific version. Could you provide some insight into the reasons behind this decision?
Thank you for your assistance.
Best regards,
shibo
1) Generally and not specific to ceph: i) If a client issues 2 overlapped write ios A then io B. If A returns first then B returns, the client knows that both ios were written to storage, but cannot assume A was written before B, it cannot assume order since they are overlapped. ii) If after A returns, the client issues io C which also returns, it can assume C was written after A. 2) The above is valid if no caching/buffering is involved in the middle. Caching can be located on the client (OS/page cache, librbd cache), or the storage server (server/hardware cache, drive cache..). Caching provides better performance but in case of crash, data in cache will be lost and it does not guarantee the same write order. Client application specify if they want to disable OS cache (direct flag) or server cache (sync flag). Clients that enable caching can issue "cache flush" at specfic times to guarantee when data is saved and have control on save order. 3) Ceph OSDs do not cache writes, there is no server cache, all i/o is considered (sync flag). If you write client app that uses librados, then rules in 1) apply If you use kernel rbd mapped devices and write with (direct flag) to avoid page cache,then rules in 1) apply. If you use librbd and disable librbd cache,then rules in 1) apply. 4) Regarding parallel behavior: concurrent i/o will execute in parallel across the many OSDs as well as within the OSD. However within the OSD, writes with the same pg number will be serialized, this is an implementation detail mainly to insure integrity of the pg metadata structures, it is not related to overlapped i/o ordering. So yes, overlapped i/o to block storage will execute in parallel. 5) I had a quick look at the changes/patches you mention, they do not alter the above. From what i understood it seems to be related to when using librbd and have librbd cache enabled, it enhances/fixes behavior of cache flushes, maybe like delaying execution of a flush if prev i/o or prev flush was inflight and not yet completed, so in case of crash we do not have image with new writes but not old. /maged