Hello, I was wondering what user experience was with using Ceph over RDMA? - How you set it up? - Documentation used to set it up? - Known issues when using it? - If you still use it? Kind regards Gabryel Mason-Williams
Hello! Mon, Oct 14, 2019 at 07:28:07AM -0000, gabryel.mason-williams wrote:
Hello,
I was wondering what user experience was with using Ceph over RDMA? - How you set it up?
We had used RoCE Lag with Mellanox ConnectX-4 Lx.
- Documentation used to set it up?
Generally, Mellanox community docs and Ceph docs: https://community.mellanox.com/s/article/bring-up-ceph-rdma---developer-s-gu...
- Known issues when using it?
Ceph's distribution does not include Systemd units with LimitMEMLOCK=infinity setting. Also it was needed to start Ceph as root to workaround some limits. Ceph rbd clients, so as mgr daemons, do not suport rdma, so it was needed to set ms_cluster_type = async+rdma ms_type = async+rdma ms_public_type = async+posix [mgr] ms_type = async+posix And we needed to disable any Jumbo Frames support in order to work with RDMA.
- If you still use it?
As I can see on my graphs, it is latency drop with Nautilus+RDMA. As for now, cluster is up and running for 2 weeks without any issues and with our production load (rbd, radosgw, cephfs).
Wow, does it really work? And why is it not supported by RBD? Can you show us the latency graphs before and after and tell the I/O pattern to which the latency applies? Previous common knowledge was that RDMA almost doesn't affect latency with Ceph, because most of the latency is in Ceph itself.
Hello!
Mon, Oct 14, 2019 at 07:28:07AM -0000, gabryel.mason-williams wrote:
Hello,
I was wondering what user experience was with using Ceph over RDMA? - How you set it up?
We had used RoCE Lag with Mellanox ConnectX-4 Lx.
- Documentation used to set it up?
Generally, Mellanox community docs and Ceph docs: https://community.mellanox.com/s/article/bring-up-ceph-rdma---developer-s-gu...
- Known issues when using it?
Ceph's distribution does not include Systemd units with LimitMEMLOCK=infinity setting. Also it was needed to start Ceph as root to workaround some limits. Ceph rbd clients, so as mgr daemons, do not suport rdma, so it was needed to set ms_cluster_type = async+rdma ms_type = async+rdma ms_public_type = async+posix [mgr] ms_type = async+posix
And we needed to disable any Jumbo Frames support in order to work with RDMA.
- If you still use it?
As I can see on my graphs, it is latency drop with Nautilus+RDMA. As for now, cluster is up and running for 2 weeks without any issues and with our production load (rbd, radosgw, cephfs). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- With best regards, Vitaliy Filippov
On 10/15/19 1:29 PM, Vitaliy Filippov wrote:
Wow, does it really work?
And why is it not supported by RBD?
Can you show us the latency graphs before and after and tell the I/O pattern to which the latency applies? Previous common knowledge was that RDMA almost doesn't affect latency with Ceph, because most of the latency is in Ceph itself.
This is still the case. RDMA might shave a bit off the network latency, but the code/CPU latency is still the highest in Ceph. There is no real benefit in using RDMA over Ethernet+IP as that latency is already very low with modern chips and switches. Wido
Hello!
Mon, Oct 14, 2019 at 07:28:07AM -0000, gabryel.mason-williams wrote:
Hello,
I was wondering what user experience was with using Ceph over RDMA? - How you set it up?
We had used RoCE Lag with Mellanox ConnectX-4 Lx.
- Documentation used to set it up?
Generally, Mellanox community docs and Ceph docs: https://community.mellanox.com/s/article/bring-up-ceph-rdma---developer-s-gu...
- Known issues when using it?
Ceph's distribution does not include Systemd units with LimitMEMLOCK=infinity setting. Also it was needed to start Ceph as root to workaround some limits. Ceph rbd clients, so as mgr daemons, do not suport rdma, so it was needed to set ms_cluster_type = async+rdma ms_type = async+rdma ms_public_type = async+posix [mgr] ms_type = async+posix
And we needed to disable any Jumbo Frames support in order to work with RDMA.
- If you still use it?
As I can see on my graphs, it is latency drop with Nautilus+RDMA. As for now, cluster is up and running for 2 weeks without any issues and with our production load (rbd, radosgw, cephfs). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
День добрий! Tue, Oct 15, 2019 at 02:29:58PM +0300, vitalif wrote:
Wow, does it really work?
And why is it not supported by RBD?
I hadn't dive into sources, but it stated in docs.
Can you show us the latency graphs before and after and tell the I/O pattern to which the latency applies? Previous common knowledge was that RDMA almost doesn't affect latency with Ceph, because most of the latency is in Ceph itself.
There is graph here. It was pure Nautilus before 10-05 and Nautilus+RDMA after. https://nc.avalon.org.ua/s/LptPTEaTeTTyKtD Link expires on Nov 1. Most of my clients is OpenStack instances with rbd volumes. Cluster consists of 30 ssd and 10 hdd osds, rbd volumes lies on ssd. It was experiment with RDMA, but it's result was resonably good to test it for a longer time.
I was wondering what user experience was with using Ceph over RDMA? - How you set it up?
We had used RoCE Lag with Mellanox ConnectX-4 Lx.
- Documentation used to set it up?
Generally, Mellanox community docs and Ceph docs: https://community.mellanox.com/s/article/bring-up-ceph-rdma---developer-s-gu...
- Known issues when using it?
Ceph's distribution does not include Systemd units with LimitMEMLOCK=infinity setting. Also it was needed to start Ceph as root to workaround some limits. Ceph rbd clients, so as mgr daemons, do not suport rdma, so it was needed to set ms_cluster_type = async+rdma ms_type = async+rdma ms_public_type = async+posix [mgr] ms_type = async+posix
And we needed to disable any Jumbo Frames support in order to work with RDMA.
- If you still use it?
As I can see on my graphs, it is latency drop with Nautilus+RDMA. As for now, cluster is up and running for 2 weeks without any issues and with our production load (rbd, radosgw, cephfs).
I don't see any changes here...
There is graph here. It was pure Nautilus before 10-05 and Nautilus+RDMA after. https://nc.avalon.org.ua/s/LptPTEaTeTTyKtD Link expires on Nov 1.
That's apply/commit latency (the exact same since BlueStore btw, no point in tracking both). It should not contain any network component. Since the path you are optimizing is inter-OSD communication: check out subop latency, that's the one where this should show up. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Tue, Oct 15, 2019 at 2:39 PM <vitalif@yourcmc.ru> wrote:
I don't see any changes here...
There is graph here. It was pure Nautilus before 10-05 and Nautilus+RDMA after. https://nc.avalon.org.ua/s/LptPTEaTeTTyKtD Link expires on Nov 1.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi All - I did some investigation into Ceph RDMA as part of a performance analysis project working with Ceph over Omnipath and NVME. I wrote up some of the analysis here: https://www.stackhpc.com/ceph-on-the-brain-a-year-with-the-human-brain-proje... <https://www.stackhpc.com/ceph-on-the-brain-a-year-with-the-human-brain-project.html> My conclusion at the time was that Ceph’s RDMA support was not portable across different RDMA-capable network fabrics, but that RoCE worked pretty well. Unfortunately, on the hardware I had available for RoCE testing the network was not the bottleneck, so I didn’t see any compelling advantage. It would be great to do this testing again on a system with the potential to really shine. This work concluded about a year ago, so might be a little out of date. Best wishes, Stig
On 15 Oct 2019, at 13:46, Paul Emmerich <paul.emmerich@croit.io> wrote:
That's apply/commit latency (the exact same since BlueStore btw, no point in tracking both). It should not contain any network component.
Since the path you are optimizing is inter-OSD communication: check out subop latency, that's the one where this should show up.
Paul
-- Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
On Tue, Oct 15, 2019 at 2:39 PM <vitalif@yourcmc.ru> wrote:
I don't see any changes here...
There is graph here. It was pure Nautilus before 10-05 and Nautilus+RDMA after. https://nc.avalon.org.ua/s/LptPTEaTeTTyKtD Link expires on Nov 1.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
gabryel.mason-williams＠diamond.ac.uk
-
Max Krasilnikov
-
Paul Emmerich
-
Stig Telfer
-
vitalif＠yourcmc.ru
-
Vitaliy Filippov
-
Wido den Hollander