Received-SPF: pass (google.com: domain of linux-kernel+bounces-119818-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: Frank Li <Frank.Li@nxp.com>
To: helgaas@kernel.org
Cc: Frank.Li@nxp.com,
	bhelgaas@google.com,
	cassel@kernel.org,
	gustavo.pimentel@synopsys.com,
	imx@lists.linux.dev,
	jdmason@kudzu.us,
	jingoohan1@gmail.com,
	kw@linux.com,
	linux-kernel@vger.kernel.org,
	linux-pci@vger.kernel.org,
	lpieralisi@kernel.org,
	mani@kernel.org,
	niklas.cassel@wdc.com,
	robh@kernel.org
Subject: [PATCH v3 1/1] PCI: dwc: Fix index 0 incorrectly being interpreted as a free ATU slot
Date: Tue, 26 Mar 2024 15:35:40 -0400
Message-Id: <20240326193540.3610570-1-Frank.Li@nxp.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?SPzjWGER6qD9g7KQKmcr/QUkH36S8cEpDGmtaoCxqyNYTUHr1ZLVuRz12/ya?=
 =?us-ascii?Q?uyyrYWRx/yPajyxlEqa+wVMwa6V5VzGJ7CphoeacafoNYcCUvRyNZx36vldk?=
 =?us-ascii?Q?Vid8aQiFF5eVW0qNbXUhz7EWeVHV5X+AhYSvLDQNXCneWr1yJjpV9OrI8eng?=
 =?us-ascii?Q?Mp0Nf6Ww0fI/Oy7GZ+x4GYBEctHOhqxl2MNS58S8IFbYXQlAvhlEnsQ4/2LT?=
 =?us-ascii?Q?OLV+1qoDwalJi66ORVtbGvCCpzsUAa9BeLQWdP/OKdJj6dwAKAjqpSq1lO6b?=
 =?us-ascii?Q?oEvNbeWsy1OKDKU8EVZNlIXDqJp9ltoL/DErXM1fbrXTUWiIABwN3r/Vy/Qk?=
 =?us-ascii?Q?0c7Oc1TO+Af9bQFp612ek45LkH3otK71DD3Mn80FbxSokG7kgqH4gbUeipr1?=
 =?us-ascii?Q?KygTuUvefuPrXha5Bnti0wVmoofJ5o8bfSgLTM0uOTQ/7cTamgU6+U6kc++6?=
 =?us-ascii?Q?qRqj/o68nSFQlfyaYMUJWTa9vMOw74vlXr5DasjTHRUBF9gPOTNDBAae6NzS?=
 =?us-ascii?Q?wsgrpkU4Dj8gh7+hsnEoAYh9wLNk6fPglSrMBAEAJl+eh83VKQ8nqPCIfj6h?=
 =?us-ascii?Q?h1sz25q+/xOCzZ+keiLX0nsIYgwgGgfWSwMPLsz8dLby6agYaduRhvn+Vbne?=
 =?us-ascii?Q?btaukWF+pL2OLhzwvi9ilFxzx3dZs6D4yJ5kMoy9UQ3suMwMAMWRwLhZDeFa?=
 =?us-ascii?Q?V3RaABM+64e05kDcyUQXuvIeVAGjQBm//XoT0IZxS0oUifyVi4ffrHqCXl0M?=
 =?us-ascii?Q?wWS3xQVOFaZ0bBmaIzZkLyfcilPRib9XarBIKPiMc52GMPqAnzkfJLzwtEFu?=
 =?us-ascii?Q?DixRz8FCAa6f8Z50F96BPTOaP33aKJgtOEmXK4wfEXjgbaz1Jed32F7brLp6?=
 =?us-ascii?Q?VsAreJwCWZDM0R7g7qn2J2O0ryXaczf5Cag3lP2zd+whFKx49CaYAK6rF/7Y?=
 =?us-ascii?Q?Z/u+dpJvzQAKdG9zktg1ibKSBBzWaiWtkIQO8vy984QzB/2eEN1wxaao7Xp0?=
 =?us-ascii?Q?THL1CFlhM9mpn0/Co12Xz/jqy0lBoy6EPq6Mr+BXt8w6OZWIpuQcer9XALHu?=
 =?us-ascii?Q?hdqreU/q4iLnU+lzz4Q5uzzaJxa5NmpeuWdaTFD8Gnfq2EbJ8tfuMWsfx+tT?=
 =?us-ascii?Q?LblFqmALWQ5C5Gmu3HsINAarM5uWxwSFo1X0BbTjXOVSVW9MJlA4HvhWBMFt?=
 =?us-ascii?Q?88Ltr+pcaFnrXkniuVdjOilZ5VGaux+0MQoOuPGoBYbbCSJPtUckLeI86vvc?=
 =?us-ascii?Q?crVrQlGZWEZX4Us/u2Cw752P/q1L2wa2M184smsq82IXpktKsGJbYbZTyuMv?=
 =?us-ascii?Q?RRZuBgacAXjlt3ox6Y3F8J+9JHDaxDCOX/yTuPDeVVPS74hZXP4DdoluHLIi?=
 =?us-ascii?Q?tWkLL9WvAMnRv3nIT8dHFFWUzHAjrL9H5CWAfufcQFkjsno4212qdtNO/Qwq?=
 =?us-ascii?Q?xZFEI03CHvHHpvXdFbTvqJdt6v9kyJUvGYfjhe8iIS1VS6DO1zjfK4l4xPaf?=
 =?us-ascii?Q?+xRZJjagBzvd1CfKjjgJFJxaU8YgwqfUkEo95ZXO3wODuxG8SF6c2MUhyLnB?=
 =?us-ascii?Q?XqI+FwwgXfYcmnHfxpFxpHLRejhvBj6BvXRalTUO?=
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1e424983-8838-4ed9-b0ee-08dc4dcbf603
X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB9642.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Mar 2024 19:35:58.2636
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: INyqbBJ8PnJDPH5LkyJKxHPKkQBtbExkLBVkTnw0M5awCv93F0QeOWQGhMcKH1i50z8lzTkGts+HJqusD/6TBQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1PR04MB7438

When PERST# assert and deassert happens on the PERST# supported platforms,
the both iATU0 and iATU6 will map inbound window to BAR0. DMA will access
to the area that was previously allocated (iATU0) for BAR0, instead of the
new area (iATU6) for BAR0.

The mapping between PCI BAR and iATU inbound window are maintained in the
dw_pcie_ep::bar_to_atu[] array. While allocating a new inbound iATU map for
a BAR, dw_pcie_ep_inbound_atu() API will first check for the availability
of the existing mapping in the array and if it is not found (i.e., value in
the array indexed by the BAR is found to be 0), then it will allocate a new
map value using find_first_zero_bit().

The issue here is, the existing logic failed to consider the fact that the
map value '0' is a valid value for BAR0. Because, find_first_zero_bit()
will return '0' as the map value for BAR0 (note that it returns the first
zero bit position).

Due to this, when PERST# assert + deassert happens on the PERST# supported
platforms, the inbound window allocation restarts from BAR0 and the
existing logic to find the BAR mapping will return '6' for BAR0 instead of
'0' due to the fact that it considers '0' as an invalid map value.

So fix this issue by always incrementing the map value before assigning to
bar_to_atu[] array and then decrementing it while fetching. This will make
sure that the map value '0' always represents the invalid mapping."

Reported-by: Niklas Cassel <Niklas.Cassel@wdc.com>
Closes: https://lore.kernel.org/linux-pci/ZXsRp+Lzg3x%2Fnhk3@x1-carbon/
Tested-by: Niklas Cassel <niklas.cassel@wdc.com>
Fixes: 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")
Reviewed-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Frank Li <Frank.Li@nxp.com>
---

Notes:
    Change from v2 to v3
    - Add impact in commit message
    - Add mani's detail description
    - Fix Closes link
    
    Change from v1 to v2
    - update subject
    - use free_win + 1 solution
    - still leave MAX_IATU_IN as 256. I am not sure if there are platfrom have
    256 ATU. Suppose it only use max 6 in current EP framework.
    - @Niklas, can you help test it. My platform become unstable today.

 drivers/pci/controller/dwc/pcie-designware-ep.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index 5befed2dc02b7..ba932bafdb230 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -139,7 +139,7 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, int type,
 	if (!ep->bar_to_atu[bar])
 		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
 	else
-		free_win = ep->bar_to_atu[bar];
+		free_win = ep->bar_to_atu[bar] - 1;
 
 	if (free_win >= pci->num_ib_windows) {
 		dev_err(pci->dev, "No free inbound window\n");
@@ -153,7 +153,11 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, int type,
 		return ret;
 	}
 
-	ep->bar_to_atu[bar] = free_win;
+	/*
+	 * Always increment free_win before assignment, since value 0 is used to identify
+	 * unallocated mapping.
+	 */
+	ep->bar_to_atu[bar] = free_win + 1;
 	set_bit(free_win, ep->ib_window_map);
 
 	return 0;
@@ -190,7 +194,10 @@ static void dw_pcie_ep_clear_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
 	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	enum pci_barno bar = epf_bar->barno;
-	u32 atu_index = ep->bar_to_atu[bar];
+	u32 atu_index = ep->bar_to_atu[bar] - 1;
+
+	if (!ep->bar_to_atu[bar])
+		return;
 
 	__dw_pcie_ep_reset_bar(pci, func_no, bar, epf_bar->flags);
 
-- 
2.34.1