Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <20210608184527.87018-1-olga.kornievskaia@gmail.com>
 <C9C7FA2C-1913-4332-91EA-B51F8E104728@oracle.com> <CAN-5tyFYyxYr4joR6uPR7zoFToYMmntNdkUkNWV=g33OVXFuOw@mail.gmail.com>
 <29835A59-A523-49FD-8D28-06CDC2F0E94C@oracle.com> <da738bdd35859aba7bdbed30da10df4e2d4087d2.camel@hammerspace.com>
 <1753b60c2cc23e6e650357fbf710ef4450310d76.camel@hammerspace.com>
 <CAN-5tyHsVprR9_0Q98X8Dd5zYgaX79V7R76KPSzDwQFMYL2qcA@mail.gmail.com>
 <e136a646089d862c476b555ea6b009cf19f5fadd.camel@hammerspace.com>
 <CAN-5tyG+fmfXsgT6rEURBKT3xUn5_pgwY=JzYCtFWKPpbnkzZQ@mail.gmail.com> <ba534fc8bb625870f31c086528c7f6d879b9156e.camel@hammerspace.com>
In-Reply-To: <ba534fc8bb625870f31c086528c7f6d879b9156e.camel@hammerspace.com>
From:   Olga Kornievskaia <olga.kornievskaia@gmail.com>
Date:   Tue, 8 Jun 2021 18:38:58 -0400
Message-ID: <CAN-5tyEJTfG9R7qA2JKar0mFzSNqnJT1ffavj-ik1buaZoqbAw@mail.gmail.com>
Subject: Re: [PATCH 1/1] NFSv4.1+ add trunking when server trunking detected
To:     Trond Myklebust <trondmy@hammerspace.com>
Cc:     "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
        "anna.schumaker@netapp.com" <anna.schumaker@netapp.com>,
        "chuck.lever@oracle.com" <chuck.lever@oracle.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Tue, Jun 8, 2021 at 6:31 PM Trond Myklebust <trondmy@hammerspace.com> wrote:
>
> On Tue, 2021-06-08 at 18:21 -0400, Olga Kornievskaia wrote:
> > On Tue, Jun 8, 2021 at 5:41 PM Trond Myklebust <
> > trondmy@hammerspace.com> wrote:
> > >
> > > On Tue, 2021-06-08 at 17:30 -0400, Olga Kornievskaia wrote:
> > > > On Tue, Jun 8, 2021 at 5:26 PM Trond Myklebust <
> > > > trondmy@hammerspace.com> wrote:
> > > > >
> > > > > On Tue, 2021-06-08 at 21:19 +0000, Trond Myklebust wrote:
> > > > > > On Tue, 2021-06-08 at 21:06 +0000, Chuck Lever III wrote:
> > > > > > >
> > > > > > >
> > > > > > > > On Jun 8, 2021, at 4:56 PM, Olga Kornievskaia <
> > > > > > > > olga.kornievskaia@gmail.com> wrote:
> > > > > > > >
> > > > > > > > On Tue, Jun 8, 2021 at 4:41 PM Chuck Lever III <
> > > > > > > > chuck.lever@oracle.com> wrote:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > On Jun 8, 2021, at 2:45 PM, Olga Kornievskaia <
> > > > > > > > > > olga.kornievskaia@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > From: Olga Kornievskaia <kolga@netapp.com>
> > > > > > > > > >
> > > > > > > > > > After trunking is discovered in
> > > > > > > > > > nfs4_discover_server_trunking(),
> > > > > > > > > > add the transport to the old client structure before
> > > > > > > > > > destroying
> > > > > > > > > > the new client structure (along with its transport).
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
> > > > > > > > > > ---
> > > > > > > > > > fs/nfs/nfs4client.c | 40
> > > > > > > > > > ++++++++++++++++++++++++++++++++++++++++
> > > > > > > > > > 1 file changed, 40 insertions(+)
> > > > > > > > > >
> > > > > > > > > > diff --git a/fs/nfs/nfs4client.c
> > > > > > > > > > b/fs/nfs/nfs4client.c
> > > > > > > > > > index 42719384e25f..984c851844d8 100644
> > > > > > > > > > --- a/fs/nfs/nfs4client.c
> > > > > > > > > > +++ b/fs/nfs/nfs4client.c
> > > > > > > > > > @@ -361,6 +361,44 @@ static int
> > > > > > > > > > nfs4_init_client_minor_version(struct nfs_client
> > > > > > > > > > *clp)
> > > > > > > > > >      return nfs4_init_callback(clp);
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > +static void nfs4_add_trunk(struct nfs_client *clp,
> > > > > > > > > > struct
> > > > > > > > > > nfs_client *old)
> > > > > > > > > > +{
> > > > > > > > > > +     struct sockaddr_storage clp_addr, old_addr;
> > > > > > > > > > +     struct sockaddr *clp_sap = (struct sockaddr
> > > > > > > > > > *)&clp_addr;
> > > > > > > > > > +     struct sockaddr *old_sap = (struct sockaddr
> > > > > > > > > > *)&old_addr;
> > > > > > > > > > +     size_t clp_salen, old_salen;
> > > > > > > > > > +     struct xprt_create xprt_args = {
> > > > > > > > > > +             .ident = old->cl_proto,
> > > > > > > > > > +             .net = old->cl_net,
> > > > > > > > > > +             .servername = old->cl_hostname,
> > > > > > > > > > +     };
> > > > > > > > > > +     struct nfs4_add_xprt_data xprtdata = {
> > > > > > > > > > +             .clp = old,
> > > > > > > > > > +     };
> > > > > > > > > > +     struct rpc_add_xprt_test rpcdata = {
> > > > > > > > > > +             .add_xprt_test = old->cl_mvops-
> > > > > > > > > > > session_trunk,
> > > > > > > > > > +             .data = &xprtdata,
> > > > > > > > > > +     };
> > > > > > > > > > +
> > > > > > > > > > +     if (clp->cl_proto != old->cl_proto)
> > > > > > > > > > +             return;
> > > > > > > > > > +     clp_salen = rpc_peeraddr(clp->cl_rpcclient,
> > > > > > > > > > clp_sap,
> > > > > > > > > > sizeof(clp_addr));
> > > > > > > > > > +     old_salen = rpc_peeraddr(old->cl_rpcclient,
> > > > > > > > > > old_sap,
> > > > > > > > > > sizeof(old_addr));
> > > > > > > > > > +
> > > > > > > > > > +     if (clp_addr.ss_family != old_addr.ss_family)
> > > > > > > > > > +             return;
> > > > > > > > > > +
> > > > > > > > > > +     xprt_args.dstaddr = clp_sap;
> > > > > > > > > > +     xprt_args.addrlen = clp_salen;
> > > > > > > > > > +
> > > > > > > > > > +     xprtdata.cred = nfs4_get_clid_cred(old);
> > > > > > > > > > +     rpc_clnt_add_xprt(old->cl_rpcclient,
> > > > > > > > > > &xprt_args,
> > > > > > > > > > +
> > > > > > > > > > rpc_clnt_setup_test_and_add_xprt,
> > > > > > > > > > &rpcdata);
> > > > > > > > >
> > > > > > > > > Is there an upper bound on the number of transports
> > > > > > > > > that
> > > > > > > > > are added to the NFS client's switch?
> > > > > > > >
> > > > > > > > I don't believe any limits exist right now. Why should
> > > > > > > > there
> > > > > > > > be a
> > > > > > > > limit? Are you saying that the client should limit
> > > > > > > > trunking?
> > > > > > > > While
> > > > > > > > this is not what's happening here, but say FS_LOCATION
> > > > > > > > returned
> > > > > > > > 100
> > > > > > > > ips for the server. Are you saying the client should be
> > > > > > > > limiting
> > > > > > > > how
> > > > > > > > many trunkable connections it would be establishing and
> > > > > > > > picking
> > > > > > > > just a
> > > > > > > > few addresses to try? What's happening with this patch is
> > > > > > > > that
> > > > > > > > say
> > > > > > > > there are 100mounts to 100 ips (each representing the
> > > > > > > > same
> > > > > > > > server
> > > > > > > > or
> > > > > > > > trunkable server(s)), without this patch a single
> > > > > > > > connection
> > > > > > > > is
> > > > > > > > kept,
> > > > > > > > with this patch we'll have 100 connections.
> > > > > > >
> > > > > > > The patch description needs to make this behavior more
> > > > > > > clear.
> > > > > > > It
> > > > > > > needs to explain "why" -- the body of the patch already
> > > > > > > explains
> > > > > > > "what". Can you include your last sentence in the
> > > > > > > description
> > > > > > > as
> > > > > > > a use case?
> > > > > > >
> > > > > > > As for the behavior... Seems to me that there are going to
> > > > > > > be
> > > > > > > only
> > > > > > > infinitesimal gains after the first few connections, and
> > > > > > > after
> > > > > > > that, it's going to be a lot for both sides to manage for
> > > > > > > no
> > > > > > > real
> > > > > > > gain. I think you do want to cap the total number of
> > > > > > > connections
> > > > > > > at a reasonable number, even in the FS_LOCATIONS case.
> > > > > > >
> > > > > >
> > > > > > I'd tend to agree. If you want to scale I/O then pNFS is the
> > > > > > way
> > > > > > to
> > > > > > go,
> > > > > > not vast numbers of connections to a single server.
> > > > > >
> > > > > BTW: AFAICS this patch will end up adding another connection
> > > > > every
> > > > > time
> > > > > we mount a new filesystem, whether or not a connection already
> > > > > exists
> > > > > to that IP address. That's unacceptable.
> > > >
> > > > Not in my testing. Mounts to the same server (same IP) different
> > > > export volumes lead to use of a single connection.
> > > > >
> > >
> > > Ah... Never mind. The comparison is made by
> > > rpc_clnt_setup_test_and_add_xprt(), and the address is discarded if
> > > it
> > > is already present.
> > >
> > > However why are you running trunking detection a second time on the
> > > address you've just run trunking detection on and have decided to
> > > add?
> >
> > Because that's where we determined that these are different clients
> > and are trunkable? But I guess also for the ease of re-using existing
> > code. There is no hook to create an xprt and add it to an existing
> > RPC
> > client. One can be created.
>
> Why not rpc_clnt_add_xprt()? That's what pNFS uses for this case.

That's what this code is using.

>
>
>
> --
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> trond.myklebust@hammerspace.com
>
>