From: Phil Edworthy <phil.edworthy@renesas.com>
To: "Liviu.Dudau@arm.com" <Liviu.Dudau@arm.com>
CC: "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-arm-kernel@lists.infradead.org" 
	<linux-arm-kernel@lists.infradead.org>,
        Bjorn Helgaas <bhelgaas@google.com>, Arnd Bergmann <arnd@arndb.de>,
        Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
        Magnus <magnus.damm@gmail.com>
Subject: RE: PCIe host controller behind IOMMU on ARM
Thread-Topic: PCIe host controller behind IOMMU on ARM
Thread-Index: AdEXBXpsqoonZ+nqRTu1yvYRg99bGAABv/qAAAApWPAAASaHgAAAPSUw
Date: Wed, 4 Nov 2015 15:19:13 +0000
Message-ID: <PS1PR06MB11804C3D0155E3EFB5C43C47F52A0@PS1PR06MB1180.apcprd06.prod.outlook.com>
References: <PS1PR06MB11807CBE8EB2AC6B0158D2B5F52A0@PS1PR06MB1180.apcprd06.prod.outlook.com>
 <20151104142412.GS963@e106497-lin.cambridge.arm.com>
 <PS1PR06MB11804263DD55C54C5987654EF52A0@PS1PR06MB1180.apcprd06.prod.outlook.com>
 <20151104150147.GT963@e106497-lin.cambridge.arm.com>
In-Reply-To: <20151104150147.GT963@e106497-lin.cambridge.arm.com>
Accept-Language: en-GB, en-US
Content-Language: en-US
spamdiagnosticoutput: 1:23
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Nov 2015 15:19:13.6259
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 53d82571-da19-47e4-9cb4-625a166a4a2a
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PS1PR06MB1178
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id tA4FJRuV014033
Content-Length: 2683
Lines: 60

Hi Liviu,

On 04 November 2015 15:02, Liviu wrote:
> On Wed, Nov 04, 2015 at 02:48:38PM +0000, Phil Edworthy wrote:
> > Hi Liviu,
> >
> > On 04 November 2015 14:24, Liviu wrote:
> > > On Wed, Nov 04, 2015 at 01:57:48PM +0000, Phil Edworthy wrote:
> > > > Hi,
> > > >
> > > > I am trying to hook up a PCIe host controller that sits behind an IOMMU,
> > > > but having some problems.
> > > >
> > > > I'm using the pcie-rcar PCIe host controller and it works fine without
> > > > the IOMMU, and I can attach the IOMMU to the controller such that any
> calls
> > > > to dma_alloc_coherent made by the controller driver uses the iommu_ops
> > > > version of dma_ops.
> > > >
> > > > However, I can't see how to make the endpoints to utilise the dma_ops that
> > > > the controller uses. Shouldn't the endpoints inherit the dma_ops from the
> > > > controller?
> > >
> > > No, not directly.
> > >
> > > > Any pointers for this?
> > >
> > > You need to understand the process through which a driver for endpoint get
> > > an address to be passed down to the device. Have a look at
> > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there.
> > > (Hint: EP driver needs to call dma_map_single).
> > >
> > > Also, you need to make sure that the bus address that ends up being set into
> > > the endpoint gets translated correctly by the host controller into an address
> > > that the IOMMU can then translate into physical address.
> > Sure, though since this is bog standard Intel PCIe ethernet card which works
> > fine when the IOMMU is effectively unused, I don’t think there is a problem
> > with that.
> >
> > The driver for the PCIe controller sets up the IOMMU mapping ok when I
> > do a test call to dma_alloc_coherent() in the controller's driver. i.e. when I
> > do this, it ends up in arm_iommu_alloc_attrs(), which calls
> > __iommu_alloc_buffer() and __alloc_iova().
> >
> > When an endpoint driver allocates and maps a dma coherent buffer it
> > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't.
> 
> Why do you think that? Remember that the only thing attached to the IOMMU is
> the
> host controller. The endpoint is on the PCIe bus, which gets a different
> translation
> that the IOMMU knows nothing about. If it helps you to visualise it better, think
> of the host controller as another IOMMU device. It's the ops of the host
> controller
> that should be invoked, not the IOMMU's.
Ok, that makes sense. I'll have a think and poke it a bit more...

Thanks for your comments
Phil
????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay?ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m????????????I?