Hi Robert,

Another option is if both RDMA ports are on the same card, then you can do RDMA with a bond. This does not work if you have two separate cards.

Yes, we recently talked to Mellanox and their engineers also recommend this way.

As far as your questions go, my guess would be that you would want to have the different NICs in different broadcast domains

yes, the idea was to use two public networks on two different NICs with addresses from different subnets. It is possible to set 2+ networks in Ceph configuration, but it’s unclear how Ceph is going to use this configuration.

or set up Source Based Routing and bind the source port on the connection (not the easiest, but allows you to have multiple NICs in the same broadcast domain). I don't have experience with Ceph in this type of configuration.

it’s too complicated and, frankly, when you’re trying to reach max performance, Source Based Routing is a bit from another area :-)

At the end of the all, we’re going to test bonding of two ports on same NIC.

Thank you.

-- 
Volodymyr Litovka
  "Vision without Execution is Hallucination." -- Thomas Edison

On Fri, Aug 2, 2019 at 9:41 AM Volodymyr Litovka <doka.ua@gmx.com> wrote:
Dear colleagues,

at the moment, we use Ceph in routed environment (OSPF, ECMP) and everything is ok, reliability is high and there is nothing to complain about. But for hardware reasons (to be more precise - RDMA offload), we are faced with the need to operate Ceph directly on physical interfaces.

According to documentation, "We generally recommend that dual-NIC systems either be configured with two IPs on the same network, or bonded."

Q1: Did anybody test and can explain, how Ceph will behave in first scenario (two IPs on the same network)? I think this configuration require just one statement in 'public network' (where both interfaces reside)? How it will distribute traffic between links, how it will detect link failures and how it will switchover?

Q2: Did anybody test a bit another scenario - both NICs have addresses in different networks and Ceph configuration contain two 'public networks'? Questions are same - how Ceph distributes traffic between links and how it recovers from link failures?

Thank you.

--
Volodymyr Litovka
  "Vision without Execution is Hallucination." -- Thomas Edison
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com