On Thu, 6 Jul 2023 at 12:54, Mark Nelson <mark.nelson@clyso.com> wrote:
On 7/6/23 06:02, Matthew Booth wrote:
On Wed, 5 Jul 2023 at 15:18, Mark Nelson <mark.nelson@clyso.com> wrote:
I'm sort of amazed that it gave you symbols without the debuginfo packages installed. I'll need to figure out a way to prevent that. Having said that, your new traces look more accurate to me. The thing that sticks out to me is the (slight?) amount of contention on the PWL m_lock in dispatch_deferred_writes, update_root_scheduled_ops, append_ops, append_sync_point(), etc.
I don't know if the contention around the m_lock is enough to cause an increase in 99% tail latency from 1.4ms to 5.2ms, but it's the first thing that jumps out at me. There appears to be a large number of threads (each tp_pwl thread, the io_context_pool threads, the qemu thread, and the bstore_aio thread) that all appear to have potential to contend on that lock. You could try dropping the number of tp_pwl threads from 4 to 1 and see if that changes anything. Will do. Any idea how to do that? I don't see an obvious rbd config option.
Thanks for looking into this, Matt
you thanked me too soon...it appears to be hard-coded in, so you'll have to do a custom build. :D
https://github.com/ceph/ceph/blob/main/src/librbd/cache/pwl/AbstractWriteLog...
Just to update: I have managed to test this today and it made no difference :( In general, though, unless it's something egregious are we really looking for something CPU-bound? Writes are 2 orders of magnitude slower than the underlying local disk. This has to be caused by something wildly inefficient. I have had a thought: the guest filesystem has 512 byte blocks, but the pwl filesystem has 4k blocks (on a 4k disk). Given that the test is of small writes, is there any chance that we're multiplying the number of physical writes in some pathological manner? Matt -- Matthew Booth