Received-SPF: pass (google.com: domain of linux-kernel+bounces-167770-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Fri, 3 May 2024 10:23:16 -0400
From: Frank Li <Frank.li@nxp.com>
To: cassel@kernel.org,
	Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= <kwilczynski@kernel.org>
Cc: bhelgaas@google.com, gustavo.pimentel@synopsys.com, helgaas@kernel.org,
	imx@lists.linux.dev, jdmason@kudzu.us, jingoohan1@gmail.com,
	kw@linux.com, linux-kernel@vger.kernel.org,
	linux-pci@vger.kernel.org, lpieralisi@kernel.org, mani@kernel.org,
	robh@kernel.org
Subject: Re: [PATCH v4 1/1] PCI: dwc: Fix index 0 incorrectly being
 interpreted as a free ATU slot
Message-ID: <ZjTzVNWAkNDJC3cs@lizhi-Precision-Tower-5810>
References: <20240412160841.925927-1-Frank.Li@nxp.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20240412160841.925927-1-Frank.Li@nxp.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?MzhMa2o3aktrQ2l2NFZEMlZnS1pTY3VQYllSTDgxVHcybzZSYi9URnF2REtE?=
 =?utf-8?B?K0VDZkJJUVRNOVl4S0V6cSs4U3p0VnZVQmR2SUJWWjRHVzdIbm45djVxR3Ax?=
 =?utf-8?B?WlpaYS95RGFaOVEyUGJQcmRzWUZkKzdUc3M3cldMYlFTaWxNRDRpWnh0NmZ5?=
 =?utf-8?B?d1ducTFLdVNUMzNabFhkL2t2cEF6c2Vtd3ZLN29mbnkzU2wzRXVvSUp2U0Mr?=
 =?utf-8?B?QXQ1TGtzRy9rRm9zNG55NTMxMVlVeVVwaWFEeVp5VmMzOHVuc0ttTHk3ckRS?=
 =?utf-8?B?Y1hFSTVyUFliNEwxcUZnMUZjR2RSdWtLNTlONmEvZVM5K0FGdWVxTnRMQW9Q?=
 =?utf-8?B?czdrQ3J4OEZEb2J1ZGNUbzZib1BiWnhoSkVpRFQyK0dJd0w4VzZnZGFObjlz?=
 =?utf-8?B?UldXSCtQbGxNRjh3TkJabGUxcXU0VDBXNk5CalpJVWsxNkZDWEFteWhEd051?=
 =?utf-8?B?RkYyOEtCT3k1V3YwaVNQVUVQY3N4bnBwaTFWTkZ2emYxTjJJOFoxa2dnMi81?=
 =?utf-8?B?OXJYMWpiZ25LbTFHc1drMnloWW5qTFJWR0NpSGU1YXdIWm03N3JYa2xMVzdy?=
 =?utf-8?B?VDBKQVREODFrZWtUNzNqZlg4WGlnV3hpM0dFN2xoYVBYRTBZckpGVGdJTER6?=
 =?utf-8?B?K0R1Z3FROFFHTWxpcjI1RkM1dncyajBpRmdXRVlBNkJKaXpqM3dZSEpoVW13?=
 =?utf-8?B?bURuV1ZKdkQ0dFRzMXB1eE5pY2VERDdUV3JYK1l5UmxaNWtuOGlKdzMxZTNs?=
 =?utf-8?B?YWNraUlxU3J5U2t6NDR2amI5dU9JR29wTVhOeDFHYTV5TDNrMkFiSDEwN1V6?=
 =?utf-8?B?bUlZWDV5Qm9jcEhvUEZnWUs3NkxsaEdFdzF5NmRrOTNQY3BmdlowdW5Mb2pE?=
 =?utf-8?B?UEVrV1FYeEk0d2R1dkg5TTdsL0hPWnpnQ1hNVUFKbUphbFJ3dzh0TmJHa2dz?=
 =?utf-8?B?cnYyY2Z0TDh2Y2xjazVpbitudWJFZXVic2pJenV2RTB0TUFwZlNHVFlkbnZ1?=
 =?utf-8?B?aWFXSjRESktSam9qNloydUhNZ1U2c2l4V21Xd245dlpHbTNjY00xLzBBb3pa?=
 =?utf-8?B?cllBQnkvZDRwNlRvdWMzNUdnQTIxY3FaeDc5bXlJYzl6MWIrS2tpVEdYMHRn?=
 =?utf-8?B?RHdRSnFqRDlvSkRhYjlmaDhoaDRXV040YXFNN3dYMjlDUmxoVVRHT3E3U3Rl?=
 =?utf-8?B?SEcvUExmL2lmMFN2NTFwYkRSRHJkMWhGUWtLdFZ1cW9GQTNoTHBjS2gzL0tE?=
 =?utf-8?B?cVEzM01XZVAvUmdLYmpzQktJWE4wT3h1bmU2YjlNaHRjQjBqcytjTDJWdVIv?=
 =?utf-8?B?UU5NT2hoQkFjSjc1MlM2NTBYa3p1QXVqYjg4Slk4cTNJT3ppVEZCQ2pNUDFp?=
 =?utf-8?B?SkpuMExSMXJDU1lsSEJ1b2VqK0F5REVvNmhPZXNaNW5kYVhqbzdyTHlFSUdE?=
 =?utf-8?B?QjAveVZFalpkTGlKUVZoVVlZdEx2MmlzZGVoZlQrTHllbGZ0cDB2R1UwWHVN?=
 =?utf-8?B?OUpFenpPTCt1WUFVaVFSNENoQ2FVZ3RRSkZpU3kxbVRvbmpUYmc4WEcrZE96?=
 =?utf-8?B?SW1ub0FudHBPdzhZbHM3MHkzMmZrNnk1ckc1STZwcjlWNnMvM3hJWmp3WWRQ?=
 =?utf-8?B?U2RYVEdlRVpLSklvaDVYa3NPSE1wMTlWTzlpRjBjU0V1bWZEcGlDOXZNQ1V0?=
 =?utf-8?B?L1pkSnQyYzBwV2lCckVOUE1HZkk3WHlGNStqemlrdkxuNElnVUdxVEZiemdC?=
 =?utf-8?B?dHB0cCtiWmNZOFl2YUNsczNYYTErUko1UjNDYldxcE4ybDM5d1ZRdFJuR3lH?=
 =?utf-8?B?NjcvbS9jUDlHL29OK3BlSGNReWVoZVgxdkdML2hkditWRmF1djhOaDNQK2o0?=
 =?utf-8?B?QXlyZEhSQ1dqaGtwN3RBRGRkYzJVOWtUS1ZVMFBTVVZSRnJINE1zYUpISFV5?=
 =?utf-8?B?QmdTT3Y4clFiOHphRCtielIwcStKR2ZBbkNycWJGS3hLbGFxWVc1VWluWld0?=
 =?utf-8?B?REZXbjdBcEtaZ1U4RCtTdFJVVFYvaDdCcFVPYTZrWVBCU3JJQ3p5cUhUbkxX?=
 =?utf-8?B?bEZwMzZNZzJLK090L2JrRUx5a1lIMDlRNHQvVnFENkpCTmt1UXlxVmlSdjdH?=
 =?utf-8?Q?LBt33CGufP8HPNb9YySBcI/nS?=
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1f3904d9-6e77-46fa-8108-08dc6b7c97f3
X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB9642.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 May 2024 14:23:25.0984
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Uj9ZnYv82AAzfrwZkqPXlvJy9XLrtnk9bwZp7QZYp/aUtRGnh7cTy0pPeK35AkhOnFk6IC+oz0ABmQ1vygHKoA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR04MB10050

On Fri, Apr 12, 2024 at 12:08:41PM -0400, Frank Li wrote:
> When PERST# assert and deassert happens on the PERST# supported platforms,
> the both iATU0 and iATU6 will map inbound window to BAR0. DMA will access
> to the area that was previously allocated (iATU0) for BAR0, instead of the
> new area (iATU6) for BAR0.
> 
> Right now, we dodge the bullet because both iATU0 and iATU6 should
> currently translate inbound accesses to BAR0 to the same allocated memory
> area. However, having two separate inbound mappings for the same BAR is a
> disaster waiting to happen.
> 
> The mapping between PCI BAR and iATU inbound window are maintained in the
> dw_pcie_ep::bar_to_atu[] array. While allocating a new inbound iATU map for
> a BAR, dw_pcie_ep_inbound_atu() API will first check for the availability
> of the existing mapping in the array and if it is not found (i.e., value in
> the array indexed by the BAR is found to be 0), then it will allocate a new
> map value using find_first_zero_bit().
> 
> The issue here is, the existing logic failed to consider the fact that the
> map value '0' is a valid value for BAR0. Because, find_first_zero_bit()
> will return '0' as the map value for BAR0 (note that it returns the first
> zero bit position).
> 
> Due to this, when PERST# assert + deassert happens on the PERST# supported
> platforms, the inbound window allocation restarts from BAR0 and the
> existing logic to find the BAR mapping will return '6' for BAR0 instead of
> '0' due to the fact that it considers '0' as an invalid map value.
> 
> So fix this issue by always incrementing the map value before assigning to
> bar_to_atu[] array and then decrementing it while fetching. This will make
> sure that the map value '0' always represents the invalid mapping."
> 
> Reported-by: Niklas Cassel <Niklas.Cassel@wdc.com>
> Closes: https://lore.kernel.org/linux-pci/ZXsRp+Lzg3x%2Fnhk3@x1-carbon/
> Tested-by: Niklas Cassel <niklas.cassel@wdc.com>
> Fixes: 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")
> Reviewed-by: Niklas Cassel <niklas.cassel@wdc.com>
> Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---

Anyone take care this patch? Krzysztof Wilczyński, Lorenzo?

Frank

> 
> Notes:
>     Change from v3 to v4
>     - commit message add "Right now, we dodge..."
>     - Add Reviewed-by: Manivannan Sadhasivam
>     
>     Change from v2 to v3
>     - Add impact in commit message
>     - Add mani's detail description
>     - Fix Closes link
>     
>     Change from v1 to v2
>     - update subject
>     - use free_win + 1 solution
>     - still leave MAX_IATU_IN as 256. I am not sure if there are platfrom have
>     256 ATU. Suppose it only use max 6 in current EP framework.
>     - @Niklas, can you help test it. My platform become unstable today.
> 
>  drivers/pci/controller/dwc/pcie-designware-ep.c | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
> index 5befed2dc02b7..ba932bafdb230 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> @@ -139,7 +139,7 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, int type,
>  	if (!ep->bar_to_atu[bar])
>  		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
>  	else
> -		free_win = ep->bar_to_atu[bar];
> +		free_win = ep->bar_to_atu[bar] - 1;
>  
>  	if (free_win >= pci->num_ib_windows) {
>  		dev_err(pci->dev, "No free inbound window\n");
> @@ -153,7 +153,11 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, int type,
>  		return ret;
>  	}
>  
> -	ep->bar_to_atu[bar] = free_win;
> +	/*
> +	 * Always increment free_win before assignment, since value 0 is used to identify
> +	 * unallocated mapping.
> +	 */
> +	ep->bar_to_atu[bar] = free_win + 1;
>  	set_bit(free_win, ep->ib_window_map);
>  
>  	return 0;
> @@ -190,7 +194,10 @@ static void dw_pcie_ep_clear_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
>  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  	enum pci_barno bar = epf_bar->barno;
> -	u32 atu_index = ep->bar_to_atu[bar];
> +	u32 atu_index = ep->bar_to_atu[bar] - 1;
> +
> +	if (!ep->bar_to_atu[bar])
> +		return;
>  
>  	__dw_pcie_ep_reset_bar(pci, func_no, bar, epf_bar->flags);
>  
> -- 
> 2.34.1
>