Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 12 Jan 2021 16:17:06 +0200
From:   Dan Aloni <dan@kernelim.com>
To:     Trond Myklebust <trond.myklebust@hammerspace.com>,
        Anna Schumaker <anna.schumaker@netapp.com>,
        linux-nfs@vger.kernel.org
Cc:     Jim Foraker <foraker1@llnl.gov>, Ben Woodard <woodard@redhat.com>
Subject: [RFC] NFSv3 RDMA multipath enhancements
Message-ID: <20210112141706.GA3146539@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Precedence: bulk

Hi Trond, Anna,

We currently have several field installations containing NFS and
SunRPC-related patches that greatly improve performance of NFSv3 clients
over RDMA setups, where link aggregation is not supported.

I would like work to integrate several of these changes to upstream, and
discuss their implementation. We managed to get a bandwidth of 33 GB/sec
from single node NFSv3 mount, and later around 92 GB/sec from a single
mount using further enhancements in RPC request dispatch.

The main change allows specifying multiple target IP addresses in a
single mount, that combined with nconnect and multiple floating IPs,
provides load balancing over several target nodes. This is good for
systems where load balancing is managed by moving a group of floating IP
addresses. This works especially well on RoCE setups.

The networking setup on these clients comprises of multiple RDMA network
interfaces that are connected to the same network, and each has its own
IP address.

The proposed change specifies a new `remoteports=<IP-addresses-ranges>`
mount option providing a group of IP addresses, from which `nconnect` at
sunrpc scope picks target transport address in round-robin. There's also
an accompanying `localports` parameter that allows local address bind so
that the source port is better controlled, in a way to ensure that
transports are not hogging a single local interface. So essentially,
this is a form of session trunking, that can be thought as an extension
to the existing `nconnect` parameter.

To my understanding NFSv4.x with pNFS has advanced dynamic transport
management logic along file layouts supporting stripe over file offsets,
however there are cases in which we would like to achieve good
performance even with the older protocol.

Before I adjust the patches I'm testing for v5.11, do you see other
implementation or user interface considerations I should take into
account?

Thanks

-- 
Dan Aloni