Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\))
Subject: Re: [RFC] NFSv3 RDMA multipath enhancements
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <20210112141706.GA3146539@gmail.com>
Date:   Wed, 13 Jan 2021 09:59:58 -0500
Cc:     Trond Myklebust <trond.myklebust@hammerspace.com>,
        Anna Schumaker <anna.schumaker@netapp.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Jim Foraker <foraker1@llnl.gov>,
        Ben Woodard <woodard@redhat.com>
Content-Transfer-Encoding: 7bit
Message-Id: <3AB7389B-2EA9-4894-9656-9C8512166E0E@oracle.com>
References: <20210112141706.GA3146539@gmail.com>
To:     Dan Aloni <dan@kernelim.com>
Precedence: bulk


> On Jan 12, 2021, at 9:17 AM, Dan Aloni <dan@kernelim.com> wrote:
> 
> Hi Trond, Anna,
> 
> We currently have several field installations containing NFS and
> SunRPC-related patches that greatly improve performance of NFSv3 clients
> over RDMA setups, where link aggregation is not supported.
> 
> I would like work to integrate several of these changes to upstream, and
> discuss their implementation. We managed to get a bandwidth of 33 GB/sec
> from single node NFSv3 mount, and later around 92 GB/sec from a single
> mount using further enhancements in RPC request dispatch.
> 
> The main change allows specifying multiple target IP addresses in a
> single mount, that combined with nconnect and multiple floating IPs,
> provides load balancing over several target nodes. This is good for
> systems where load balancing is managed by moving a group of floating IP
> addresses. This works especially well on RoCE setups.
> 
> The networking setup on these clients comprises of multiple RDMA network
> interfaces that are connected to the same network, and each has its own
> IP address.
> 
> The proposed change specifies a new `remoteports=<IP-addresses-ranges>`
> mount option providing a group of IP addresses, from which `nconnect` at
> sunrpc scope picks target transport address in round-robin. There's also
> an accompanying `localports` parameter that allows local address bind so
> that the source port is better controlled, in a way to ensure that
> transports are not hogging a single local interface. So essentially,
> this is a form of session trunking, that can be thought as an extension
> to the existing `nconnect` parameter.
> 
> To my understanding NFSv4.x with pNFS has advanced dynamic transport
> management logic along file layouts supporting stripe over file offsets,
> however there are cases in which we would like to achieve good
> performance even with the older protocol.

Hi Dan, my curiosity is piqued about the RPC request dispatch changes
you have in mind. Can you post them here for review?

Also, if you can tell us, what NFS server supports NFS/RDMA but not
NFSv4 ?


> Before I adjust the patches I'm testing for v5.11, do you see other
> implementation or user interface considerations I should take into
> account?
> 
> Thanks
> 
> -- 
> Dan Aloni

--
Chuck Lever