The @nents value that was passed to ib_dma_map_sg() has to be passed
to the matching ib_dma_unmap_sg() call. If ib_dma_map_sg() choses to
concatenate sg entries, it will return a different nents value than
it was passed.
The bug was exposed by recent changes to the AMD IOMMU driver.
Reported-by: Andre Tomt <[email protected]>
Suggested-by: Robin Murphy <[email protected]>
Fixes: 1f541895dae9 ("xprtrdma: Don't defer MR recovery if ro_map fails")
Signed-off-by: Chuck Lever <[email protected]>
---
net/sunrpc/xprtrdma/frwr_ops.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
Hey Andre, please try this out. It just reverts the bit of brokenness that
Robin observed this morning. I've done basic testing here with Intel
IOMMU systems, no change in behavior (ie, all good to go).
diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c
index 095be887753e..449bb51e4fe8 100644
--- a/net/sunrpc/xprtrdma/frwr_ops.c
+++ b/net/sunrpc/xprtrdma/frwr_ops.c
@@ -313,10 +313,9 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt,
break;
}
mr->mr_dir = rpcrdma_data_dir(writing);
+ mr->mr_nents = i;
- mr->mr_nents =
- ib_dma_map_sg(ia->ri_id->device, mr->mr_sg, i, mr->mr_dir);
- if (!mr->mr_nents)
+ if (!ib_dma_map_sg(ia->ri_id->device, mr->mr_sg, i, mr->mr_dir))
goto out_dmamap_err;
ibmr = mr->frwr.fr_mr;
On 11.02.2020 20:58, Chuck Lever wrote:
> The @nents value that was passed to ib_dma_map_sg() has to be passed
> to the matching ib_dma_unmap_sg() call. If ib_dma_map_sg() choses to
> concatenate sg entries, it will return a different nents value than
> it was passed.
>
> The bug was exposed by recent changes to the AMD IOMMU driver.
This seems to fail differently on my system; mount fails with:
mount.nfs: mount system call failed
and the kernel log reports:
[ 38.890344] NFS: Registering the id_resolver key type
[ 38.890351] Key type id_resolver registered
[ 38.890352] Key type id_legacy registered
[ 38.901799] NFS: nfs4_discover_server_trunking unhandled error -5.
Exiting with error EIO
[ 38.901817] NFS4: Couldn't follow remote path
amd_iommu=off still works
One detail I accidentally left out of the original report is that the
server (intel system) is running Ubuntu 20.04 ("beta") userspace, and
AMD clients are Ubuntu 19.10 userspace. Although I dont believe this to
matter at this point.
> Reported-by: Andre Tomt <[email protected]>
> Suggested-by: Robin Murphy <[email protected]>
> Fixes: 1f541895dae9 ("xprtrdma: Don't defer MR recovery if ro_map fails")
> Signed-off-by: Chuck Lever <[email protected]>
> ---
> net/sunrpc/xprtrdma/frwr_ops.c | 5 ++---
> 1 file changed, 2 insertions(+), 3 deletions(-)
>
> Hey Andre, please try this out. It just reverts the bit of brokenness that
> Robin observed this morning. I've done basic testing here with Intel
> IOMMU systems, no change in behavior (ie, all good to go).
>
> diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c
> index 095be887753e..449bb51e4fe8 100644
> --- a/net/sunrpc/xprtrdma/frwr_ops.c
> +++ b/net/sunrpc/xprtrdma/frwr_ops.c
> @@ -313,10 +313,9 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt,
> break;
> }
> mr->mr_dir = rpcrdma_data_dir(writing);
> + mr->mr_nents = i;
>
> - mr->mr_nents =
> - ib_dma_map_sg(ia->ri_id->device, mr->mr_sg, i, mr->mr_dir);
> - if (!mr->mr_nents)
> + if (!ib_dma_map_sg(ia->ri_id->device, mr->mr_sg, i, mr->mr_dir))
> goto out_dmamap_err;
>
> ibmr = mr->frwr.fr_mr;
>
>
Hi Andre, thanks for trying this out.
> On Feb 11, 2020, at 3:50 PM, Andre Tomt <[email protected]> wrote:
>
> On 11.02.2020 20:58, Chuck Lever wrote:
>> The @nents value that was passed to ib_dma_map_sg() has to be passed
>> to the matching ib_dma_unmap_sg() call. If ib_dma_map_sg() choses to
>> concatenate sg entries, it will return a different nents value than
>> it was passed.
>> The bug was exposed by recent changes to the AMD IOMMU driver.
>
> This seems to fail differently on my system; mount fails with:
> mount.nfs: mount system call failed
>
> and the kernel log reports:
> [ 38.890344] NFS: Registering the id_resolver key type
> [ 38.890351] Key type id_resolver registered
> [ 38.890352] Key type id_legacy registered
> [ 38.901799] NFS: nfs4_discover_server_trunking unhandled error -5. Exiting with error EIO
> [ 38.901817] NFS4: Couldn't follow remote path
>
> amd_iommu=off still works
>
> One detail I accidentally left out of the original report is that the server (intel system) is running Ubuntu 20.04 ("beta") userspace, and AMD clients are Ubuntu 19.10 userspace. Although I dont believe this to matter at this point.
Next thing to try:
# trace-cmd record -e sunrpc -e rpcrdma
then issue the mount command. Once it completes, ^C the trace-cmd and send me trace.dat.
Try this with both the v5.4 kernel and the v5.5 kernel (and note that trace-cmd overwrites trace.dat, so copy it out between tests).
>> Reported-by: Andre Tomt <[email protected]>
>> Suggested-by: Robin Murphy <[email protected]>
>> Fixes: 1f541895dae9 ("xprtrdma: Don't defer MR recovery if ro_map fails")
>> Signed-off-by: Chuck Lever <[email protected]>
>> ---
>> net/sunrpc/xprtrdma/frwr_ops.c | 5 ++---
>> 1 file changed, 2 insertions(+), 3 deletions(-)
>> Hey Andre, please try this out. It just reverts the bit of brokenness that
>> Robin observed this morning. I've done basic testing here with Intel
>> IOMMU systems, no change in behavior (ie, all good to go).
>> diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c
>> index 095be887753e..449bb51e4fe8 100644
>> --- a/net/sunrpc/xprtrdma/frwr_ops.c
>> +++ b/net/sunrpc/xprtrdma/frwr_ops.c
>> @@ -313,10 +313,9 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt,
>> break;
>> }
>> mr->mr_dir = rpcrdma_data_dir(writing);
>> + mr->mr_nents = i;
>> - mr->mr_nents =
>> - ib_dma_map_sg(ia->ri_id->device, mr->mr_sg, i, mr->mr_dir);
>> - if (!mr->mr_nents)
>> + if (!ib_dma_map_sg(ia->ri_id->device, mr->mr_sg, i, mr->mr_dir))
>> goto out_dmamap_err;
>> ibmr = mr->frwr.fr_mr;
>
--
Chuck Lever
On 11.02.2020 23:00, Chuck Lever wrote:
> Hi Andre, thanks for trying this out.
>
>> On Feb 11, 2020, at 3:50 PM, Andre Tomt <[email protected]> wrote:
>>
>> On 11.02.2020 20:58, Chuck Lever wrote:
>>> The @nents value that was passed to ib_dma_map_sg() has to be passed
>>> to the matching ib_dma_unmap_sg() call. If ib_dma_map_sg() choses to
>>> concatenate sg entries, it will return a different nents value than
>>> it was passed.
>>> The bug was exposed by recent changes to the AMD IOMMU driver.
>>
>> This seems to fail differently on my system; mount fails with:
>> mount.nfs: mount system call failed
>>
>> and the kernel log reports:
>> [ 38.890344] NFS: Registering the id_resolver key type
>> [ 38.890351] Key type id_resolver registered
>> [ 38.890352] Key type id_legacy registered
>> [ 38.901799] NFS: nfs4_discover_server_trunking unhandled error -5. Exiting with error EIO
>> [ 38.901817] NFS4: Couldn't follow remote path
>>
>> amd_iommu=off still works
>>
>> One detail I accidentally left out of the original report is that the server (intel system) is running Ubuntu 20.04 ("beta") userspace, and AMD clients are Ubuntu 19.10 userspace. Although I dont believe this to matter at this point.
>
> Next thing to try:
>
> # trace-cmd record -e sunrpc -e rpcrdma
>
> then issue the mount command. Once it completes, ^C the trace-cmd and send me trace.dat.
>
> Try this with both the v5.4 kernel and the v5.5 kernel (and note that trace-cmd overwrites trace.dat, so copy it out between tests).
I've uploaded them to https://tomt.net/temp/rdmaiommubug/
I'll probably do a 5.5.3 with the v1 fix as well, should show up
momentarily.
>>> Reported-by: Andre Tomt <[email protected]>
>>> Suggested-by: Robin Murphy <[email protected]>
>>> Fixes: 1f541895dae9 ("xprtrdma: Don't defer MR recovery if ro_map fails")
>>> Signed-off-by: Chuck Lever <[email protected]>
>>> ---
>>> net/sunrpc/xprtrdma/frwr_ops.c | 5 ++---
>>> 1 file changed, 2 insertions(+), 3 deletions(-)
>>> Hey Andre, please try this out. It just reverts the bit of brokenness that
>>> Robin observed this morning. I've done basic testing here with Intel
>>> IOMMU systems, no change in behavior (ie, all good to go).
>>> diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c
>>> index 095be887753e..449bb51e4fe8 100644
>>> --- a/net/sunrpc/xprtrdma/frwr_ops.c
>>> +++ b/net/sunrpc/xprtrdma/frwr_ops.c
>>> @@ -313,10 +313,9 @@ struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt,
>>> break;
>>> }
>>> mr->mr_dir = rpcrdma_data_dir(writing);
>>> + mr->mr_nents = i;
>>> - mr->mr_nents =
>>> - ib_dma_map_sg(ia->ri_id->device, mr->mr_sg, i, mr->mr_dir);
>>> - if (!mr->mr_nents)
>>> + if (!ib_dma_map_sg(ia->ri_id->device, mr->mr_sg, i, mr->mr_dir))
>>> goto out_dmamap_err;
>>> ibmr = mr->frwr.fr_mr;
>>
>
> --
> Chuck Lever
>
>
>