Received-SPF: pass (google.com: domain of linux-kernel+bounces-139182-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Wed, 10 Apr 2024 14:16:52 -0400
From: Frank Li <Frank.li@nxp.com>
To: Manivannan Sadhasivam <mani@kernel.org>
Cc: Niklas Cassel <cassel@kernel.org>, helgaas@kernel.org,
	bhelgaas@google.com, gustavo.pimentel@synopsys.com,
	imx@lists.linux.dev, jdmason@kudzu.us, jingoohan1@gmail.com,
	kw@linux.com, linux-kernel@vger.kernel.org,
	linux-pci@vger.kernel.org, lpieralisi@kernel.org, robh@kernel.org
Subject: Re: [PATCH v3 1/1] PCI: dwc: Fix index 0 incorrectly being
 interpreted as a free ATU slot
Message-ID: <ZhbXlOMNxc2nMIW8@lizhi-Precision-Tower-5810>
References: <20240326193540.3610570-1-Frank.Li@nxp.com>
 <ZhPFmFYorWa-sfLp@ryzen>
 <20240410180341.GF16629@thinkpad>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20240410180341.GF16629@thinkpad>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?UHAvYUZLd0hWZ2lra1Bjb1pzTHZNekNKZkN2aXRoVkdIK211bytGRVdmamdM?=
 =?utf-8?B?SVczSTJzQURaenc2YVhud1grVkxJcCtWVlN6UzgvVXZYeWFyNDhlTW1WM2Jj?=
 =?utf-8?B?UlFUMVpteXhpRGdYUnZMakF6SCtVazNWSm1HUUkwL000V3IrakFRbEczYTlS?=
 =?utf-8?B?OURQZXZ1Y1NZMlNPZlhrZllWWi9WRjFoRWRjWHZDcDh3Ui9HVFEyQm1UbFR1?=
 =?utf-8?B?cUowYU1GQXFybHFSeDEzNjlWb2JQZFZXb05rOFZ2TTVzUHliWkRXWlFJZjY0?=
 =?utf-8?B?VndFcFFQcVNoZzVFcDZrdjJqSGlpbnpsV2JCT2lrOS9raDU1YmU0d0xRdks3?=
 =?utf-8?B?Vjd1SUl4cDZ0SUlqZ0xURHlsb2N2Mnd3ZnAzdlBnSU0yZ1BJTHNGNXR3K1JC?=
 =?utf-8?B?alJoRXRKOTV2NHlYNm9RbkFESlJjR2tIbTBHMHhpYjV3akZ5UnJ3UzArU2RR?=
 =?utf-8?B?ak8vcVhWcjd6d3VqeDhSMmhzQklZUmV0WHZsMjAwbXV6SDlIWjh5VGdiYkxm?=
 =?utf-8?B?MDBOU3B1TmhuZjlYblpmT0RVMVdnTW5oT1J2RG5UNmFLZWZUYXR3MlgvWmJG?=
 =?utf-8?B?TGsyOTRIL1JqblFrZE1KOXdtaTFXMmdDT3pTYjBmMHI4bmU1TmVLN2dkMHZR?=
 =?utf-8?B?VEZ0K0VaWTdDNnYyUVVxOE5FemlXa2FRc2N0YS9WKzA4ZWw0a0tzTkNDeWZN?=
 =?utf-8?B?TXhXWTVqanJmVk1SVGJ1WFg0d1V5QVBCKzZ0cjN6MVR5WTRUaExqTjJZTlNt?=
 =?utf-8?B?WHpTblRQNGt1SDN3Vitqd0FXWFlYbzZvbkYxTDZsZkwyMWJRaXV1Ylo0MDdm?=
 =?utf-8?B?OTdzWDJwU2lWVTM2VnZjbFNUTXFNcUc1TlhKY3Qyb2c4RGdSc2pBUjZtSnlp?=
 =?utf-8?B?L1pNTmd6dll4dFY3WHVXV20yRW1BTUpjZnBLSEgzY1o3dEhNSjRReUlrM2pD?=
 =?utf-8?B?ZVlZV1FxZTFuZzdXU0JCUXBXalM4NEdUMWhmVDM5STFzNXVna25md3UvdDJj?=
 =?utf-8?B?REUrVXhMWi9OOCtrNWliT2REdHJKYnBlOTczNmJpVDBMTXdQTzFqZ01LU3Jq?=
 =?utf-8?B?RFNneUM4NWVOS0JwdUVacEVDd09DTVRvNWRUMGJKUk90TU9kUHh3MHJpOHN4?=
 =?utf-8?B?MFFxQXZ0WkI4cEZxeG5ZVi9PTENKTUFZVS9EN1FUakdpaVpPTGxVbTJaOThy?=
 =?utf-8?B?K3JpMWxiUE9rZEhWSEF2QlpoSzE5eXArclZmRXpyM1FJSUt0R2VHNjBJdDE0?=
 =?utf-8?B?aHhidjRHTC95SFpwdXd3bVlOeFZjV29SUEo4elY4eEI2TTB3dnZoL2JYd1pp?=
 =?utf-8?B?b2lHb213S2VsTk56eTA1ak13QmNaVmhOZ0tvajc4Q2hoRFM3Y0ZETnI4cGZS?=
 =?utf-8?B?RHVaYXhjS0twd2IrL3owckgrVTdLaDJ1R21EZ2VJcmhKcHdodFJZL0tkYzFr?=
 =?utf-8?B?R3llTXRScmFlU0JpSjVYQTcxL3dwckNxQWp4cEl2WTd6elBnRktDeUtwUFlM?=
 =?utf-8?B?Y1A5VWdUaVBuenMzeHZpTG1JZjdETk4vc2RiTXlEWVFIWFY2Vko2U0NTOXp0?=
 =?utf-8?B?ci9JUkc5aml3TWFBQjNrWmdnK0p6dm56WFE1ZHZvOCtMN3REVDY0dE1QMVU5?=
 =?utf-8?B?Vzg0SnZ6dm1HVGVlN1dKd2pqbjREV1dIUkZVRXdiNFc2N051ZXk3N3JkRTM5?=
 =?utf-8?B?ZnQzb1pVNHNockJ4WEQ1SHlxQ0JGSXNrRW9ZQ09qWE9IeGk4VDNWUkNya0lo?=
 =?utf-8?B?SXIvOExDY3JBT0lkS0Rlc0tTQnlpRnI5cGJBbW56RWxRNEloVkN5TUdzaC9G?=
 =?utf-8?B?RzJWNWhOOC9YQUtldm1QMTZPYUpUY1RXY2oxK2RLRHRxZDNWdER1NkJ4NExk?=
 =?utf-8?B?Q2o3cHV1Qy9mSlg1SzhzUEU2Q2xCNm1TZUFYSTFNa3NuOFVVT21ZMDdzeTVX?=
 =?utf-8?B?QzBLdzVrZGlVMXFzSkVWcHhwU29FVUZuY2x2S2taVXkvYW55am83b1ZYUHRB?=
 =?utf-8?B?d3NqQ25WMEV2c1pxek9oRUVvS2xKMnNiVHVJV0RkV2syWHg0VSt3WXRSK2NN?=
 =?utf-8?B?Lys4VlJHVWRDQ2pWeDZJOXEwdWpnWHVYR3NhZVJOaC9IblpEeVhvOXdTS2o0?=
 =?utf-8?Q?o1tAbSKusS8ajc+Eox4tn6WB8?=
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4fac13af-7abe-42fa-61d8-08dc598a6a8f
X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB9642.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Apr 2024 18:17:00.8877
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: eIWHLQcv/Ajh7txHr3DR1Gh6OWQ2qhIdr02UXhyiStRBsK//kHEWXIDK+kVY9F/cJXAIk0k8tCr8sw9FYyPtfQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR04MB8090

On Wed, Apr 10, 2024 at 11:33:41PM +0530, Manivannan Sadhasivam wrote:
> On Mon, Apr 08, 2024 at 12:23:20PM +0200, Niklas Cassel wrote:
> > On Tue, Mar 26, 2024 at 03:35:40PM -0400, Frank Li wrote:
> > > When PERST# assert and deassert happens on the PERST# supported platforms,
> > > the both iATU0 and iATU6 will map inbound window to BAR0. DMA will access
> > > to the area that was previously allocated (iATU0) for BAR0, instead of the
> > > new area (iATU6) for BAR0.
> > 
> > Nit: If we want additional clarity, we could also add:
> > ""
> > Right now, we dodge the bullet because both iATU0 and iATU6 should currently
> > translate inbound accesses to BAR0 to the same allocated memory area. However,
> > having two separate inbound mappings for the same BAR is a disaster waiting to
> > happen.
> > ""
> 
> Since Bjorn asked for the above info, it should get added.
> 
> With that,
> 
> Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>

Mani:

Do you need me rework patch? Or you can handle it by yourself when apply?

Frank

> 
> - Mani
> 
> > 
> > If the maintainers feel like this additional information is important, I think
> > it could be added while applying. (But I also think that the existing commit
> > message is detailed enough to be applied as is.)
> > 
> > 
> > > 
> > > The mapping between PCI BAR and iATU inbound window are maintained in the
> > > dw_pcie_ep::bar_to_atu[] array. While allocating a new inbound iATU map for
> > > a BAR, dw_pcie_ep_inbound_atu() API will first check for the availability
> > > of the existing mapping in the array and if it is not found (i.e., value in
> > > the array indexed by the BAR is found to be 0), then it will allocate a new
> > > map value using find_first_zero_bit().
> > > 
> > > The issue here is, the existing logic failed to consider the fact that the
> > > map value '0' is a valid value for BAR0. Because, find_first_zero_bit()
> > > will return '0' as the map value for BAR0 (note that it returns the first
> > > zero bit position).
> > > 
> > > Due to this, when PERST# assert + deassert happens on the PERST# supported
> > > platforms, the inbound window allocation restarts from BAR0 and the
> > > existing logic to find the BAR mapping will return '6' for BAR0 instead of
> > > '0' due to the fact that it considers '0' as an invalid map value.
> > > 
> > > So fix this issue by always incrementing the map value before assigning to
> > > bar_to_atu[] array and then decrementing it while fetching. This will make
> > > sure that the map value '0' always represents the invalid mapping."
> > > 
> > > Reported-by: Niklas Cassel <Niklas.Cassel@wdc.com>
> > > Closes: https://lore.kernel.org/linux-pci/ZXsRp+Lzg3x%2Fnhk3@x1-carbon/
> > > Tested-by: Niklas Cassel <niklas.cassel@wdc.com>
> > > Fixes: 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")
> > > Reviewed-by: Niklas Cassel <niklas.cassel@wdc.com>
> > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > ---
> > > 
> > > Notes:
> > >     Change from v2 to v3
> > >     - Add impact in commit message
> > >     - Add mani's detail description
> > >     - Fix Closes link
> > >     
> > >     Change from v1 to v2
> > >     - update subject
> > >     - use free_win + 1 solution
> > >     - still leave MAX_IATU_IN as 256. I am not sure if there are platfrom have
> > >     256 ATU. Suppose it only use max 6 in current EP framework.
> > >     - @Niklas, can you help test it. My platform become unstable today.
> > > 
> > >  drivers/pci/controller/dwc/pcie-designware-ep.c | 13 ++++++++++---
> > >  1 file changed, 10 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > > index 5befed2dc02b7..ba932bafdb230 100644
> > > --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> > > +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > > @@ -139,7 +139,7 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, int type,
> > >  	if (!ep->bar_to_atu[bar])
> > >  		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
> > >  	else
> > > -		free_win = ep->bar_to_atu[bar];
> > > +		free_win = ep->bar_to_atu[bar] - 1;
> > >  
> > >  	if (free_win >= pci->num_ib_windows) {
> > >  		dev_err(pci->dev, "No free inbound window\n");
> > > @@ -153,7 +153,11 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, int type,
> > >  		return ret;
> > >  	}
> > >  
> > > -	ep->bar_to_atu[bar] = free_win;
> > > +	/*
> > > +	 * Always increment free_win before assignment, since value 0 is used to identify
> > > +	 * unallocated mapping.
> > > +	 */
> > > +	ep->bar_to_atu[bar] = free_win + 1;
> > >  	set_bit(free_win, ep->ib_window_map);
> > >  
> > >  	return 0;
> > > @@ -190,7 +194,10 @@ static void dw_pcie_ep_clear_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
> > >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > >  	enum pci_barno bar = epf_bar->barno;
> > > -	u32 atu_index = ep->bar_to_atu[bar];
> > > +	u32 atu_index = ep->bar_to_atu[bar] - 1;
> > > +
> > > +	if (!ep->bar_to_atu[bar])
> > > +		return;
> > >  
> > >  	__dw_pcie_ep_reset_bar(pci, func_no, bar, epf_bar->flags);
> > >  
> > > -- 
> > > 2.34.1
> > > 
> > 
> 
> -- 
> மணிவண்ணன் சதாசிவம்