Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
Subject: Re: [PATCH RFC] svcrdma: Ignore source port when computing DRC hash
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <9E0019E1-1C1B-465C-B2BF-76372029ABD8@talpey.com>
Date:   Wed, 5 Jun 2019 13:25:03 -0400
Cc:     linux-rdma@vger.kernel.org,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <955993A4-0626-4819-BC6F-306A50E2E048@oracle.com>
References: <20190605121518.2150.26479.stgit@klimt.1015granger.net>
 <9E0019E1-1C1B-465C-B2BF-76372029ABD8@talpey.com>
To:     Tom Talpey <tom@talpey.com>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

Hi Tom-

> On Jun 5, 2019, at 12:43 PM, Tom Talpey <tom@talpey.com> wrote:
>=20
> On 6/5/2019 8:15 AM, Chuck Lever wrote:
>> The DRC is not working at all after an RPC/RDMA transport reconnect.
>> The problem is that the new connection uses a different source port,
>> which defeats DRC hash.
>>=20
>> An NFS/RDMA client's source port is meaningless for RDMA transports.
>> The transport layer typically sets the source port value on the
>> connection to a random ephemeral port. The server already ignores it
>> for the "secure port" check. See commit 16e4d93f6de7 ("NFSD: Ignore
>> client's source port on RDMA transports").
>=20
> Where does the entropy come from, then, for the server to not
> match other requests from other mount points on this same client?

The first ~200 bytes of each RPC Call message.

[ Note that this has some fun ramifications for calls with small
RPC headers that use Read chunks. ]


> Any time an XID happens to match on a second mount, it will trigger
> incorrect server processing, won't it?

Not a risk for clients that use only a single transport per
client-server pair.


> And since RDMA is capable of
> such high IOPS, the likelihood seems rather high.

Only when the server's durable storage is slow enough to cause
some RPC requests to have extremely high latency.

And, most clients use an atomic counter for their XIDs, so they
are also likely to wrap that counter over some long-pending RPC
request.

The only real answer here is NFSv4 sessions.


> Missing the cache
> might actually be safer than hitting, in this case.

Remember that _any_ retransmit on RPC/RDMA requires a fresh
connection, that includes NFSv3, to reset credit accounting
due to the lost half of the RPC Call/Reply pair.

I can very quickly reproduce bad (non-deterministic) behavior
by running a software build on an NFSv3 on RDMA mount point
with disconnect injection. If the DRC issue is addressed, the
software build runs to completion.

IMO we can't leave things the way they are.


> Tom.
>=20
>> I'm not sure why I never noticed this before.
>>=20
>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>> Cc: stable@vger.kernel.org
>> ---
>>  net/sunrpc/xprtrdma/svc_rdma_transport.c |    7 ++++++-
>>  1 file changed, 6 insertions(+), 1 deletion(-)
>>=20
>> diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c =
b/net/sunrpc/xprtrdma/svc_rdma_transport.c
>> index 027a3b0..1b3700b 100644
>> --- a/net/sunrpc/xprtrdma/svc_rdma_transport.c
>> +++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c
>> @@ -211,9 +211,14 @@ static void handle_connect_req(struct rdma_cm_id =
*new_cma_id,
>>  	/* Save client advertised inbound read limit for use later in =
accept. */
>>  	newxprt->sc_ord =3D param->initiator_depth;
>>=20
>> -	/* Set the local and remote addresses in the transport */
>>  	sa =3D (struct sockaddr =
*)&newxprt->sc_cm_id->route.addr.dst_addr;
>>  	svc_xprt_set_remote(&newxprt->sc_xprt, sa, svc_addr_len(sa));
>> +	/* The remote port is arbitrary and not under the control of the
>> +	 * ULP. Set it to a fixed value so that the DRC continues to =
work
>> +	 * after a reconnect.
>> +	 */
>> +	rpc_set_port((struct sockaddr *)&newxprt->sc_xprt.xpt_remote, =
0);
>> +
>>  	sa =3D (struct sockaddr =
*)&newxprt->sc_cm_id->route.addr.src_addr;
>>  	svc_xprt_set_local(&newxprt->sc_xprt, sa, svc_addr_len(sa));
>>=20
>>=20
>>=20
>>=20

--
Chuck Lever