Hi All - I did some investigation into Ceph RDMA as part of a performance analysis project working with Ceph over Omnipath and NVME. I wrote up some of the analysis here: https://www.stackhpc.com/ceph-on-the-brain-a-year-with-the-human-brain-proje... <https://www.stackhpc.com/ceph-on-the-brain-a-year-with-the-human-brain-project.html> My conclusion at the time was that Ceph’s RDMA support was not portable across different RDMA-capable network fabrics, but that RoCE worked pretty well. Unfortunately, on the hardware I had available for RoCE testing the network was not the bottleneck, so I didn’t see any compelling advantage. It would be great to do this testing again on a system with the potential to really shine. This work concluded about a year ago, so might be a little out of date. Best wishes, Stig
On 15 Oct 2019, at 13:46, Paul Emmerich <paul.emmerich@croit.io> wrote:
That's apply/commit latency (the exact same since BlueStore btw, no point in tracking both). It should not contain any network component.
Since the path you are optimizing is inter-OSD communication: check out subop latency, that's the one where this should show up.
Paul
-- Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
On Tue, Oct 15, 2019 at 2:39 PM <vitalif@yourcmc.ru> wrote:
I don't see any changes here...
There is graph here. It was pure Nautilus before 10-05 and Nautilus+RDMA after. https://nc.avalon.org.ua/s/LptPTEaTeTTyKtD Link expires on Nov 1.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io