Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
Subject: Re: [PATCH v1] xprtrdma: Fix DMA scatter-gather list mapping
 imbalance
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <8781b043-74a6-4ee0-d1c9-46f797b4aec2@tomt.net>
Date:   Tue, 11 Feb 2020 17:00:48 -0500
Cc:     robin.murphy@arm.com,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        iommu@lists.linux-foundation.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <DC2EEB01-5C19-4FB5-B103-D9ADF09E07FA@oracle.com>
References: <158145102079.515252.3226617475691911684.stgit@morisot.1015granger.net>
 <8781b043-74a6-4ee0-d1c9-46f797b4aec2@tomt.net>
To:     Andre Tomt <andre@tomt.net>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

Hi Andre, thanks for trying this out.

> On Feb 11, 2020, at 3:50 PM, Andre Tomt <andre@tomt.net> wrote:
>=20
> On 11.02.2020 20:58, Chuck Lever wrote:
>> The @nents value that was passed to ib_dma_map_sg() has to be passed
>> to the matching ib_dma_unmap_sg() call. If ib_dma_map_sg() choses to
>> concatenate sg entries, it will return a different nents value than
>> it was passed.
>> The bug was exposed by recent changes to the AMD IOMMU driver.
>=20
> This seems to fail differently on my system; mount fails with:
> mount.nfs: mount system call failed
>=20
> and the kernel log reports:
> [   38.890344] NFS: Registering the id_resolver key type
> [   38.890351] Key type id_resolver registered
> [   38.890352] Key type id_legacy registered
> [   38.901799] NFS: nfs4_discover_server_trunking unhandled error -5. =
Exiting with error EIO
> [   38.901817] NFS4: Couldn't follow remote path
>=20
> amd_iommu=3Doff still works
>=20
> One detail I accidentally left out of the original report is that the =
server (intel system) is running Ubuntu 20.04 ("beta") userspace, and =
AMD clients are Ubuntu 19.10 userspace. Although I dont believe this to =
matter at this point.

Next thing to try:

# trace-cmd record -e sunrpc -e rpcrdma

then issue the mount command. Once it completes, ^C the trace-cmd and =
send me trace.dat.

Try this with both the v5.4 kernel and the v5.5 kernel (and note that =
trace-cmd overwrites trace.dat, so copy it out between tests).


>> Reported-by: Andre Tomt <andre@tomt.net>
>> Suggested-by: Robin Murphy <robin.murphy@arm.com>
>> Fixes: 1f541895dae9 ("xprtrdma: Don't defer MR recovery if ro_map =
fails")
>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>> ---
>>  net/sunrpc/xprtrdma/frwr_ops.c |    5 ++---
>>  1 file changed, 2 insertions(+), 3 deletions(-)
>> Hey Andre, please try this out. It just reverts the bit of brokenness =
that
>> Robin observed this morning. I've done basic testing here with Intel
>> IOMMU systems, no change in behavior (ie, all good to go).
>> diff --git a/net/sunrpc/xprtrdma/frwr_ops.c =
b/net/sunrpc/xprtrdma/frwr_ops.c
>> index 095be887753e..449bb51e4fe8 100644
>> --- a/net/sunrpc/xprtrdma/frwr_ops.c
>> +++ b/net/sunrpc/xprtrdma/frwr_ops.c
>> @@ -313,10 +313,9 @@ struct rpcrdma_mr_seg *frwr_map(struct =
rpcrdma_xprt *r_xprt,
>>  			break;
>>  	}
>>  	mr->mr_dir =3D rpcrdma_data_dir(writing);
>> +	mr->mr_nents =3D i;
>>  -	mr->mr_nents =3D
>> -		ib_dma_map_sg(ia->ri_id->device, mr->mr_sg, i, =
mr->mr_dir);
>> -	if (!mr->mr_nents)
>> +	if (!ib_dma_map_sg(ia->ri_id->device, mr->mr_sg, i, mr->mr_dir))
>>  		goto out_dmamap_err;
>>    	ibmr =3D mr->frwr.fr_mr;
>=20

--
Chuck Lever