Received-SPF: pass (google.com: domain of linux-kernel+bounces-199717-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Mon, 3 Jun 2024 17:04:23 -0400
From: Frank Li <Frank.li@nxp.com>
To: Robin Murphy <robin.murphy@arm.com>
Cc: Bjorn Helgaas <helgaas@kernel.org>, Richard Zhu <hongxing.zhu@nxp.com>,
	Lucas Stach <l.stach@pengutronix.de>,
	Lorenzo Pieralisi <lpieralisi@kernel.org>,
	Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= <kw@linux.com>,
	Rob Herring <robh@kernel.org>, Bjorn Helgaas <bhelgaas@google.com>,
	Shawn Guo <shawnguo@kernel.org>,
	Sascha Hauer <s.hauer@pengutronix.de>,
	Pengutronix Kernel Team <kernel@pengutronix.de>,
	Fabio Estevam <festevam@gmail.com>,
	NXP Linux Team <linux-imx@nxp.com>,
	Philipp Zabel <p.zabel@pengutronix.de>,
	Liam Girdwood <lgirdwood@gmail.com>,
	Mark Brown <broonie@kernel.org>,
	Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>,
	Krzysztof Kozlowski <krzysztof.kozlowski+dt@linaro.org>,
	Conor Dooley <conor+dt@kernel.org>, linux-pci@vger.kernel.org,
	imx@lists.linux.dev, linux-arm-kernel@lists.infradead.org,
	linux-kernel@vger.kernel.org, bpf@vger.kernel.org,
	devicetree@vger.kernel.org, Will Deacon <will@kernel.org>,
	Joerg Roedel <joro@8bytes.org>, Jason Gunthorpe <jgg@ziepe.ca>,
	Alyssa Rosenzweig <alyssa@rosenzweig.io>,
	Marc Zyngier <maz@kernel.org>
Subject: Re: [PATCH v5 08/12] PCI: imx6: Config look up table(LUT) to support
 MSI ITS and IOMMU for i.MX95
Message-ID: <Zl4v10Od99et+tLX@lizhi-Precision-Tower-5810>
References: <20240603171921.GA685838@bhelgaas>
 <3d24fecf-1fdb-4804-9a51-d6c34a9d65c6@arm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <3d24fecf-1fdb-4804-9a51-d6c34a9d65c6@arm.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?2kGxLojdLpFSe4C4HI/68D43PyJz+LB4jz/lKuDao0REaBxHFU0zFxTBe93H?=
 =?us-ascii?Q?vFFjHuixDKgCJMl4kVeo0nQjb+Bun83086n9L03tUIsCEtFDzomQjTmq/yQP?=
 =?us-ascii?Q?jYSVu5Wuo7aeZv6VzHm9z7Oi6wYk31u83RsNnPaSVAb24o5uPVNCDR27swXo?=
 =?us-ascii?Q?56+oYcJAzmJFnkp655wkTwvWF4RgvM8cVbOr/u9xkgZQ/eSI7RKlyuI6k7u+?=
 =?us-ascii?Q?7E6OeghYxYCP3ZYTsjNW52yJRaLBOEZ90sRFht7DiMCWBryto8MLk0idunBv?=
 =?us-ascii?Q?k3nulIsyLujJ1QGAqXXc4WD54iGeVgSAB+aR9Xcyo59Vap/kvNZyJM5NPIgG?=
 =?us-ascii?Q?WYylFJq72nTDaso82IXaLwRlQiY8cvKkUNQQapdEKPcXZCo0wHd4VmVAmAxZ?=
 =?us-ascii?Q?dackmAEShDI7nlT+QtqxPjo2ZsjqyRzQWTpljIBhUO7JmleDPSucOpCI9Oty?=
 =?us-ascii?Q?1dVuOG0UxdPdx36+fmJpF5NKxNzlpUWJgaFzMBf6MWXYj2wmwNcESahh45kf?=
 =?us-ascii?Q?uNsCekbawEMZlOAizWjJHAbP81Ksts/Z0SSDCVMd9zdlQ+LIUzpBcFDQ29SK?=
 =?us-ascii?Q?Ic9N8WBXoI8tI6k4LlhganTjAg1o/SKosfSeX/QZ27gYtnSg/ACHsO6naeCE?=
 =?us-ascii?Q?ZtDlA4Vt8cEYAK/nJ66ZDOEF96czh4uPKAneMsVE6nLiFZVz2gTmCJDVXl1I?=
 =?us-ascii?Q?jxnGbXTTHwOUy2v7rYvdT+P72NPnJgPDuEU6XJLnFWiJ0QxZ5rU5HK2vJe/l?=
 =?us-ascii?Q?hUsIRn2uCpeeprN+ry9S/3x6zFn7y2boVeZnsMFfJVYN/nLbzwnj8JUJoYJh?=
 =?us-ascii?Q?7r2wm016D8chitUCGTwIynlM5mZZWTaBS57gIxKTDY+YngA/weg5tdYGwL0I?=
 =?us-ascii?Q?7JUdhmgzvwJA3cd39B2smr7HF0hPaE9+XnhDpX0usqIcMnD28/t11MtE1u0g?=
 =?us-ascii?Q?Phltf/tTQlF1upZR8yrwlFmAdaMOTJ59dpYMNc/ZTyKRK3ApCFnFUQMxWBLD?=
 =?us-ascii?Q?j1GEFu967EsCMROew0BL0ba50R2E/+7jofEdcvrL0IzmBDzsxFN8ovtEVvUa?=
 =?us-ascii?Q?j1TTCYIue+KSB0b5p8JAAG8KE86GaXwtZFOZkTQnUbeorhzUuEzZfyFofD3S?=
 =?us-ascii?Q?UWvlTmd+akniu3cOxWBpxnifO5aCwTWsKTyZ6Da1F++Ijez3B/lH62gKo5Fw?=
 =?us-ascii?Q?CKke8ez917vU2+uPgaRjPGUFhKNANaWAGGLMHKRk4lLQ5kJUJQAgoQiozEKJ?=
 =?us-ascii?Q?kF4M1vxVHwy/W0czqYOpwblxs/n7jwAJbAmPOArDHyq+07THFvwQw25RBajR?=
 =?us-ascii?Q?y7Y5GvwEWkdGDCa2iiP8+JF6dG8Jj03MJFdAm0CpDsWCDMkVTfKVtHn/ZOsD?=
 =?us-ascii?Q?w4K0t0OJ0HT53OWUHlQENMBpOGgNk6FC9ltnA55Y2msyY+by2rg1VS+ye7G/?=
 =?us-ascii?Q?sTlWiukOLu7/w3Lp+7aGtlKNvIfyNlJyaG/J8T8DZ1ghf3HDDq8JnUxUdjha?=
 =?us-ascii?Q?q+VSda8PY7jl/meXtH9nDBkArqFU7ado/VVqdbQpOqoVzzoaRrVBpqYgjT+e?=
 =?us-ascii?Q?hPl0TuOnCWEwRfalYnpy3NnFXMB5fAKabZQDpTGf?=
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 82ae940a-1c12-489a-0556-08dc8410c5ad
X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB9642.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2024 21:04:35.1621
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: trdia4jcSzRB36n8nf/BFBP3HzoO2d7bUAG09lkWB42VQ/7HrfdMSSk2ebuNci2vG2WpHaxf+uHzzwRV0XFGgw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR04MB9635

On Mon, Jun 03, 2024 at 09:20:03PM +0100, Robin Murphy wrote:
> On 2024-06-03 6:19 pm, Bjorn Helgaas wrote:
> > On Fri, May 31, 2024 at 03:58:49PM +0100, Robin Murphy wrote:
> > > On 2024-05-31 12:08 am, Bjorn Helgaas wrote:
> > > > [+cc IOMMU and pcie-apple.c folks for comment]
> > > > 
> > > > On Tue, May 28, 2024 at 03:39:21PM -0400, Frank Li wrote:
> > > > > For the i.MX95, configuration of a LUT is necessary to convert Bus Device
> > > > > Function (BDF) to stream IDs, which are utilized by both IOMMU and ITS.
> > > > > This involves examining the msi-map and smmu-map to ensure consistent
> > > > > mapping of PCI BDF to the same stream IDs. Subsequently, LUT-related
> > > > > registers are configured. In the absence of an msi-map, the built-in MSI
> > > > > controller is utilized as a fallback.
> > > > > 
> > > > > Additionally, register a PCI bus notifier to trigger imx_pcie_add_device()
> > > > > upon the appearance of a new PCI device and when the bus is an iMX6 PCI
> > > > > controller. This function configures the correct LUT based on Device Tree
> > > > > Settings (DTS).
> > > > 
> > > > This scheme is pretty similar to apple_pcie_bus_notifier().  If we
> > > > have to do this, I wish it were *more* similar, i.e., copy the
> > > > function names, bitmap tracking, code structure, etc.
> > > > 
> > > > I don't really know how stream IDs work, but I assume they are used on
> > > > most or all arm64 platforms, so I'm a little surprised that of all the
> > > > PCI host drivers used on arm64, only pcie-apple.c and pci-imx6.c need
> > > > this notifier.
> > > 
> > > This is one of those things that's mostly at the mercy of the PCIe root
> > > complex implementation. Typically the SMMU StreamID and/or GIC ITS DeviceID
> > > is derived directly from the PCI RID, sometimes with additional high-order
> > > bits hard-wired to disambiguate PCI segments. I believe this RID-translation
> > > LUT is a particular feature of the the Synopsys IP - I know there's also one
> > > on the NXP Layerscape platforms, but on those it's programmed by the
> > > bootloader, which also generates the appropriate "msi-map" and "iommu-map"
> > > properties to match. Ideally that's what i.MX should do as well, but hey.
> > 
> > Maybe this RID-translation is a feature of i.MX, not of Synopsys?  I
> > see that the LUT CSR accesses use IMX95_* definitions.
> 
> Well, it's not unreasonable to call things "IMX95" in this context if they
> are only relevant to the configuration used by i.MX95, and not to the other
> i.MX SoCs which this driver also supports. However the data register fields
> certainly look suspiciously similar to those used on Layerscape[1], although
> I guess that still doesn't rule out it being NXP's own widget either.
> Anyway, the exact details aren't really significant, the point was really
> just to say don't expect this to generalise much beyond what you've seen
> already, and that there's precedent for bootloaders doing this for us.

It is re-used layerscape design at this point. I don't know the history,
why choose use bootloader to config it, supposed it should be done at
kernel. We found some problem by using bootloader to config LUT. Most
layserscape system PCI devices are fixed during boot.

During I debug layerscape PCI, I was trapped by uboot many times. It is
quite anoise, especially using difference version uboot.

> 
> > > If it's really necessary to do this programming from Linux, then there's
> > > still no point in it being dynamic - the mappings cannot ever change, since
> > > the rest of the kernel believes that what the DT said at boot time was
> > > already a property of the hardware. It would be a lot more logical, and
> > > likely simpler, for the driver to just read the relevant map property and
> > > program the entire LUT to match, all in one go at controller probe time.
> > > Rather like what's already commonly done with the parsing of "dma-ranges" to
> > > program address-translation LUTs for inbound windows.
> > > 
> > > Plus that would also give a chance of safely dealing with bad DTs specifying
> > > invalid ID mappings (by refusing to probe at all). As it is, returning an
> > > error from a child's BUS_NOTIFY_ADD_DEVICE does nothing except prevent any
> > > further notifiers from running at that point - the device will still be
> > > added, allowed to bind a driver, and able to start sending DMA/MSI traffic
> > > without the controller being correctly programmed, which at best won't work
> > > and at worst may break the whole system.
> > 
> > Frank, could the imx LUT be programmed once at boot-time instead of at
> > device-add time?  I'm guessing maybe not because apparently there is a
> > risk of running out of LUT entries?
> 
> The risk still exists just as much either way - if we have a bogus DT and/or
> just more PCI RIDs present than we can handle, we're going to have a bad
> time. There's no advantage to only finding that out once we try to add the
> 33rd device and it's too late to even do anything about it.
> 
> In fact if anything, this notifier approach exacerbates that risk the most
> by consuming one LUT entry per PCI RID regardless of whether an
> "iommu-map-mask" is involved. Assuming the IMX95_PE0_LUT_MASK field is the
> same as its Layerscape counterpart, we could support >32 RIDs if the map and
> mask are constructed to squash multiple RIDs onto each StreamID (the SMMU
> driver supports this), and we have the up-front information to easily
> configure hardware masking in the LUT itself. It's not necessarily possible
> to reconstruct such mappings from only seeing individual input and output
> values one-by-one.

iommu may share one stream id for multi-devices. but ITS MSI can't. each
device's MSI index start from 0. It needs difference stream id for each
device.

> 
> Thanks,
> Robin.
> 
> [1] https://source.denx.de/u-boot/u-boot/-/blob/master/drivers/pci/pcie_layerscape_fixup.c?ref_type=heads#L83

Thanks, but It can't resolve device hot-plug problem.

> 
> > It sounds like the consequences of running out of LUT entries are
> > catastrophic, e.g., memory corruption from mis-directed DMA?  If
> > that's possible, I think we need to figure out how to prevent the
> > device from being used, not just dev_warn() about it.
> > 
> > Bjorn