Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
Subject: Re: AMD IOMMU stops RDMA NFS from working since kernel 5.5 (bisected)
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <21c801a6-9a8b-1ebb-7e41-76e8385116ea@arm.com>
Date:   Tue, 11 Feb 2020 11:03:39 -0500
Cc:     Andre Tomt <andre@tomt.net>, Tom Murphy <tmurphy@arista.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Joerg Roedel <jroedel@suse.de>,
        iommu@lists.linux-foundation.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <A411A8A6-ECEF-4EAD-84A1-99A30A213D8E@oracle.com>
References: <7ee099af-e6bb-18fe-eb93-2a8abd401570@tomt.net>
 <20200211072537.GD23114@suse.de>
 <2CE039F4-3519-4481-B0E2-840D24EE4428@oracle.com>
 <ac758665-9127-9a52-4f03-49fecc5289a2@arm.com>
 <3507674A-F860-4B65-BD46-93431DD268AC@oracle.com>
 <21c801a6-9a8b-1ebb-7e41-76e8385116ea@arm.com>
To:     Robin Murphy <robin.murphy@arm.com>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk


> On Feb 11, 2020, at 10:32 AM, Robin Murphy <robin.murphy@arm.com> =
wrote:
>=20
> On 11/02/2020 3:24 pm, Chuck Lever wrote:
>>> On Feb 11, 2020, at 10:12 AM, Robin Murphy <robin.murphy@arm.com> =
wrote:
>>>=20
>>> On 11/02/2020 1:48 pm, Chuck Lever wrote:
>>>> Andre-
>>>> Thank you for the detailed report!
>>>> Tom-
>>>> There is a rich set of trace points available in the RPC/RDMA =
implementation in 5.4/5.5, fwiw.
>>>> Please keep me in the loop, let me know if there is anything I can =
do to help.
>>>=20
>>> One aspect that may be worth checking is whether there's anywhere =
that assumes a successful return value from dma_map_sg() is always the =
same as the number of entries passed in - that's the most obvious way =
the iommu-dma code differs (legitimately) from the previous amd-iommu =
implementation.
>> net/sunrpc/xprtrdma/frwr_ops.c: frwr_map()
>> 317         mr->mr_nents =3D
>> 318                 ib_dma_map_sg(ia->ri_id->device, mr->mr_sg, i, =
mr->mr_dir);
>> 319         if (!mr->mr_nents)
>> 320                 goto out_dmamap_err;
>> Should that rather be "if (mr->mr_nents !=3D i)" ?
>=20
> No, that much is OK - the point is that dma_map_sg() may pack the DMA =
addresses such that sg_dma_len(sg) > sg->length - however, subsequently =
passing that mr->nents to dma_unmap_sg() in frwr_mr_recycle() (rather =
than the original value of i) looks at a glance like an example of how =
things may start to get out-of-whack.

Robin, your explanation makes sense to me. I can post a fix for this =
imbalance later today for Andre to try.


--
Chuck Lever