Hi All - 

I did some investigation into Ceph RDMA as part of a performance analysis project working with Ceph over Omnipath and NVME.  

I wrote up some of the analysis here: https://www.stackhpc.com/ceph-on-the-brain-a-year-with-the-human-brain-project.html

My conclusion at the time was that Ceph’s RDMA support was not portable across different RDMA-capable network fabrics, but that RoCE worked pretty well.  Unfortunately, on the hardware I had available for RoCE testing the network was not the bottleneck, so I didn’t see any compelling advantage.  It would be great to do this testing again on a system with the potential to really shine.

This work concluded about a year ago, so might be a little out of date.

Best wishes,
Stig


On 15 Oct 2019, at 13:46, Paul Emmerich <paul.emmerich@croit.io> wrote:

That's apply/commit latency (the exact same since BlueStore btw, no
point in tracking both). It should not contain any network component.

Since the path you are optimizing is inter-OSD communication: check
out subop latency, that's the one where this should show up.


Paul

--
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

On Tue, Oct 15, 2019 at 2:39 PM <vitalif@yourcmc.ru> wrote:

I don't see any changes here...

There is graph here. It was pure Nautilus before 10-05 and
Nautilus+RDMA after.
https://nc.avalon.org.ua/s/LptPTEaTeTTyKtD
Link expires on Nov 1.
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io