Thanks for the thinking. By 'traffic' I mean: when a user space rbd write has as a destination three replica osds in the same chassis
eek.
does the whole write get shipped out to the mon and then back
Mons are control-plane only.
All the 'usual suspects' like lossy ethernets and miswirings, etc. have been checked. It's actually painful to sit and wait while 'update-initramfs' can take over a minute when the vm is chassis-local to the osds getting the write info.
You have shared almost none of your hardware or use-case. We know that you’re doing convergence, with unspecified CPU, memory, drives. We also don’t know how heavy your colocated compute workload is. Since you mention update-initramfs, I’ll guess that your workload is VMs with RBD volumes attached to libvirt/QEMU? With unspecified RBD cache configuration. We also know nothing of your network setup and saturation. I have to suspect that either you’re doing something fundamentally wrong, or should just set up a RAID6 volume and carve out LVMs.