Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
Subject: Re: [PATCH 0/9] Multiple network connections for a single NFS mount.
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <87muj3tuuk.fsf@notabene.neil.brown.name>
Date:   Fri, 31 May 2019 09:46:32 -0400
Cc:     Olga Kornievskaia <aglo@umich.edu>,
        Anna Schumaker <Anna.Schumaker@netapp.com>,
        Trond Myklebust <trondmy@hammerspace.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <4316E30B-1BD7-4F0E-8375-03E9F85FFD2B@oracle.com>
References: <155917564898.3988.6096672032831115016.stgit@noble.brown>
 <001DED71-0E0D-46B1-BA34-84E6ACCBB79F@oracle.com>
 <87muj3tuuk.fsf@notabene.neil.brown.name>
To:     NeilBrown <neilb@suse.com>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk


> On May 30, 2019, at 6:56 PM, NeilBrown <neilb@suse.com> wrote:
>=20
> On Thu, May 30 2019, Chuck Lever wrote:
>=20
>> Hi Neil-
>>=20
>> Thanks for chasing this a little further.
>>=20
>>=20
>>> On May 29, 2019, at 8:41 PM, NeilBrown <neilb@suse.com> wrote:
>>>=20
>>> This patch set is based on the patches in the multipath_tcp branch =
of
>>> git://git.linux-nfs.org/projects/trondmy/nfs-2.6.git
>>>=20
>>> I'd like to add my voice to those supporting this work and wanting =
to
>>> see it land.
>>> We have had customers/partners wanting this sort of functionality =
for
>>> years.  In SLES releases prior to SLE15, we've provide a
>>> "nosharetransport" mount option, so that several filesystem could be
>>> mounted from the same server and each would get its own TCP
>>> connection.
>>=20
>> Is it well understood why splitting up the TCP connections result
>> in better performance?
>>=20
>>=20
>>> In SLE15 we are using this 'nconnect' feature, which is much nicer.
>>>=20
>>> Partners have assured us that it improves total throughput,
>>> particularly with bonded networks, but we haven't had any concrete
>>> data until Olga Kornievskaia provided some concrete test data - =
thanks
>>> Olga!
>>>=20
>>> My understanding, as I explain in one of the patches, is that =
parallel
>>> hardware is normally utilized by distributing flows, rather than
>>> packets.  This avoid out-of-order deliver of packets in a flow.
>>> So multiple flows are needed to utilizes parallel hardware.
>>=20
>> Indeed.
>>=20
>> However I think one of the problems is what happens in simpler =
scenarios.
>> We had reports that using nconnect > 1 on virtual clients made things
>> go slower. It's not always wise to establish multiple connections
>> between the same two IP addresses. It depends on the hardware on each
>> end, and the network conditions.
>=20
> This is a good argument for leaving the default at '1'.  When
> documentation is added to nfs(5), we can make it clear that the =
optimal
> number is dependant on hardware.

Is there any visibility into the NIC hardware that can guide this =
setting?


>> What about situations where the network capabilities between server =
and
>> client change? Problem is that neither endpoint can detect that; TCP
>> usually just deals with it.
>=20
> Being able to manually change (-o remount) the number of connections
> might be useful...

Ugh. I have problems with the administrative interface for this feature,
and this is one of them.

Another is what prevents your client from using a different nconnect=3D
setting on concurrent mounts of the same server? It's another case of a
per-mount setting being used to control a resource that is shared across
mounts.

Adding user tunables has never been known to increase the aggregate
amount of happiness in the universe. I really hope we can come up with
a better administrative interface... ideally, none would be best.


>> Related Work:
>>=20
>> We now have protocol (more like conventions) for clients to discover
>> when a server has additional endpoints so that it can establish
>> connections to each of them.
>>=20
>> https://datatracker.ietf.org/doc/rfc8587/
>>=20
>> and
>>=20
>> =
https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rfc5661-msns-update/
>>=20
>> Boiled down, the client uses fs_locations and trunking detection to
>> figure out when two IP addresses are the same server instance.
>>=20
>> This facility can also be used to establish a connection over a
>> different path if network connectivity is lost.
>>=20
>> There has also been some exploration of MP-TCP. The magic happens
>> under the transport socket in the network layer, and the RPC client
>> is not involved.
>=20
> I would think that SCTP would be the best protocol for NFS to use as =
it
> supports multi-streaming - several independent streams.  That would
> require that hardware understands it of course.
>=20
> Though I have examined MP-TCP closely, it looks like it is still fully
> sequenced, so it would be tricky for two RPC messages to be assembled
> into TCP frames completely independently - at least you would need
> synchronization on the sequence number.
>=20
> Thanks for your thoughts,
> NeilBrown
>=20
>=20
>>=20
>>=20
>>> Comments most welcome.  I'd love to see this, or something similar,
>>> merged.
>>>=20
>>> Thanks,
>>> NeilBrown
>>>=20
>>> ---
>>>=20
>>> NeilBrown (4):
>>>     NFS: send state management on a single connection.
>>>     SUNRPC: enhance rpc_clnt_show_stats() to report on all xprts.
>>>     SUNRPC: add links for all client xprts to debugfs
>>>=20
>>> Trond Myklebust (5):
>>>     SUNRPC: Add basic load balancing to the transport switch
>>>     SUNRPC: Allow creation of RPC clients with multiple connections
>>>     NFS: Add a mount option to specify number of TCP connections to =
use
>>>     NFSv4: Allow multiple connections to NFSv4.x servers
>>>     pNFS: Allow multiple connections to the DS
>>>     NFS: Allow multiple connections to a NFSv2 or NFSv3 server
>>>=20
>>>=20
>>> fs/nfs/client.c                      |    3 +
>>> fs/nfs/internal.h                    |    2 +
>>> fs/nfs/nfs3client.c                  |    1=20
>>> fs/nfs/nfs4client.c                  |   13 ++++-
>>> fs/nfs/nfs4proc.c                    |   22 +++++---
>>> fs/nfs/super.c                       |   12 ++++
>>> include/linux/nfs_fs_sb.h            |    1=20
>>> include/linux/sunrpc/clnt.h          |    1=20
>>> include/linux/sunrpc/sched.h         |    1=20
>>> include/linux/sunrpc/xprt.h          |    1=20
>>> include/linux/sunrpc/xprtmultipath.h |    2 +
>>> net/sunrpc/clnt.c                    |   98 =
++++++++++++++++++++++++++++++++--
>>> net/sunrpc/debugfs.c                 |   46 ++++++++++------
>>> net/sunrpc/sched.c                   |    3 +
>>> net/sunrpc/stats.c                   |   15 +++--
>>> net/sunrpc/sunrpc.h                  |    3 +
>>> net/sunrpc/xprtmultipath.c           |   23 +++++++-
>>> 17 files changed, 204 insertions(+), 43 deletions(-)
>>>=20
>>> --
>>> Signature
>>>=20
>>=20
>> --
>> Chuck Lever

--
Chuck Lever