Received-SPF: pass (google.com: domain of linux-kernel+bounces-90952-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Mon, 4 Mar 2024 11:47:36 -0500
From: Frank Li <Frank.li@nxp.com>
To: Manivannan Sadhasivam <mani@kernel.org>
Cc: Niklas Cassel <Niklas.Cassel@wdc.com>,
	"imx@lists.linux.dev" <imx@lists.linux.dev>,
	Jingoo Han <jingoohan1@gmail.com>,
	Gustavo Pimentel <gustavo.pimentel@synopsys.com>,
	Lorenzo Pieralisi <lpieralisi@kernel.org>,
	Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= <kw@linux.com>,
	Rob Herring <robh@kernel.org>, Bjorn Helgaas <bhelgaas@google.com>,
	Jon Mason <jdmason@kudzu.us>,
	"open list:PCI DRIVER FOR SYNOPSYS DESIGNWARE" <linux-pci@vger.kernel.org>,
	open list <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 1/1] PCI: dwc: Fix BAR0 wrong map to iATU6 after root
 complex reinit endpoint
Message-ID: <ZeX7KDuwLWxwb5Kw@lizhi-Precision-Tower-5810>
References: <20231219044844.1195294-1-Frank.Li@nxp.com>
 <ZYFrUWM7JXdv7rtb@x1-carbon>
 <ZYGmpaf18pJgM/qj@lizhi-Precision-Tower-5810>
 <ZYGq6RdCfdhXFF/9@x1-carbon>
 <20240304084841.GJ2647@thinkpad>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20240304084841.GJ2647@thinkpad>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?TlJtck9PK0hBYjlmWk9odDk4SDFETVBrbWExYUMza1gvdDJIZkxFOEdSWm9H?=
 =?utf-8?B?QzJoVGh2aFc2OURoS240Rm1OdktkTXA5b3BLTnovY085aGloY01MSWlYQmpu?=
 =?utf-8?B?NnVsSjMzSlZqV2s1NU1kK2s0QkYvRk5GTis1STVRRElJWVdJNVZYVWR3Tldt?=
 =?utf-8?B?dDBnLzVjTHR2cGVaNUhCMTR2TE1DbGxTN21ZQ0tKbEFkSkhaS2lBVDdlMmcw?=
 =?utf-8?B?OVdvQXZ4TXZlclFOK21oU2FwdnNPNGFDY2VKNndQd2VOMEpLUG9wQ3pqZzNY?=
 =?utf-8?B?b3QxUnRwUkVQYWpBQlBhV1pJRHNYcTlLMmFlaW5FVVZVOHduZEh1V2g5VWRU?=
 =?utf-8?B?WEFHeUs1a21CSDZkdWszd0wvNldRRTVIRGY1eUFwNE5CQ3NVckVLc0YzNVlv?=
 =?utf-8?B?ZGE3L0FBbXAvQjFFNjhyMElDaUFoNk44OW02bm5SdFNyL2dUem0yakxub2JP?=
 =?utf-8?B?UEhGaUlSZHJFNE42blMrMlU2UU1WdmJmb1JrZkozcTRYK0lBWFlhYmRHekJi?=
 =?utf-8?B?TmRYcCtNc2E5Nk0wNmNXdUNGTlpiZisxcGxEdzVHdDZCTVdyS0dYMDQ5SG1N?=
 =?utf-8?B?b0lHMUR0VjV6RmdOR0EzT25RUU1RNWRoWHFYRXM4WXE2NDVxaTRGYUlrQm1r?=
 =?utf-8?B?OE5EMDhpY1ZzVytYWExLdkxPcFpMZ2dlUXBoMnFOZmwvWFgwWXpucUFWUExQ?=
 =?utf-8?B?Q1FITktXVDBBZ281RjNMOEdSekd3azJrNi9PV0JGMlJsbWRGUUhoU044TDcv?=
 =?utf-8?B?c25ta0tlMzZOMHN0bGphM1BOV1lOcDZtZUhVSkdqZ3JzR1k5NkJRT3NlZzBu?=
 =?utf-8?B?QjlyVEl4Z3dlbWZhMkhCTmw1QWdZOEE5ZkdnWjV2K2xtQXVGRGFVOWtUSEY4?=
 =?utf-8?B?eGhLclUwUTB1UFQrMzM4K1ZuTGFSR0F2TE9IVEwyK1FxZHVCS1o2SW5mVmlC?=
 =?utf-8?B?dzIvbjd4VHF2aW9FT3lCNXhjSjVOUW1MSEl1ZmgxRHkvdjBNS1gwd3dMWHEx?=
 =?utf-8?B?cnRIWkdIZGZWTTYxYVVVK3NhaTlBU3FKRzVQTEpoUWw3aWxwQlRiVzBQVUZG?=
 =?utf-8?B?cGgzQWxQT3dFVWM3OE1PckQ0cFdBWTlzc0Y4WTFlUlNtUVZoZmtZRk41TXpN?=
 =?utf-8?B?L2NZdXF3MW9HTVIyZ3B1cEdka3o2UGFpQ0ZLZkV4RTNIVDNSd0traHdJVUU5?=
 =?utf-8?B?ZzkzcERmODJ2Y2d5Q1Z5SnpzLytWZzIzMFo1SzdBb0wxSFo2UVRjZEJ5RHJX?=
 =?utf-8?B?aS91MGsrNEVod0hxaFZMSGlFaUZTZFVqM0VPcVRGSEUzdzRndDVWbDFLRGxk?=
 =?utf-8?B?UXJ4WnRPZXNtaUVTSEpSVTljMkZ0UjN6QUZsT01jYU9yN3NNQkkycjVKV2Ja?=
 =?utf-8?B?TENGMlBPTG9pQnpzVWFyemNVcXkyRSs2anpTZ0NWc2FFWWppRHJjTUtXVlJm?=
 =?utf-8?B?cS9vemRxdHpiMVM4ZTE2QzQrVm11V0RhNWRkdDR6YldSeWEyeDFVbndvQlNa?=
 =?utf-8?B?TldmQ2RSUGZMT2RBV2JwaHpXb0NLOFMrRzV6VlBJcXBVc0RzS3RhUHRCcDdt?=
 =?utf-8?B?YmFlQVVrNlFLaVI4ZFZCTWsxeW5BekV1RnhKL1BMY2hmZ3QyMWV5NFluL1FW?=
 =?utf-8?B?YTRWTm5VbWlsOWhQRTFiZS9hVDVOSjQ4cHRyaUFrMmxoVXI1RWwyVVpJQnI5?=
 =?utf-8?B?cUNqcmxKVlZMRHJHdGN4TDM3Qm0zaWVPUmg0ZGVLTFU4OWZwYnNTUUZIMjRU?=
 =?utf-8?B?ZVgyVk9LQXp0OGxHVFZiNXpOUWwxQXlJK3hYazM5TmlDMjJpNWcvS2M3cnIw?=
 =?utf-8?B?QlZTdGM4T2RTVzN3cUtZTlY5dTB4Zm91dzNDNWJYc2NxQnIyVUZBcGhKcmty?=
 =?utf-8?B?VDRtblNZa29iS2JIUEZuMiszbFVRVDdNUHFYQzNUalhpTkQ2ZHIyT2liR05h?=
 =?utf-8?B?Ryt1VlkwQWY5alVhMmlONmRoSFRiRWdTYUt6dmEvZE52eW9QSUc5c1loZ25I?=
 =?utf-8?B?b3RNdmE3Q0k2UVVwWE5nRytRSVdBUTQ0b3VpeFQydjVvYi81a1p6ZUtkNmZm?=
 =?utf-8?B?STV5aTNCY1daek1INm1obCtKKzBmai84K0owMzlMYzlPMkFGb0FmeWdWRW1l?=
 =?utf-8?Q?CIBp49YsWkEyU0kQPqzOo6qLl?=
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d3d23100-8e1c-4b93-4fd4-08dc3c6ad072
X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB9642.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2024 16:47:44.2534
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Wzxbe/cEjJF0knlP3RKm7ysM+QTWy7fliu5KPKBcFUWMEyVgzXKbUo0o4SN57KN0y3OEfT/YYRQD/av30NScpg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU2PR04MB8520

On Mon, Mar 04, 2024 at 02:18:41PM +0530, Manivannan Sadhasivam wrote:
> On Tue, Dec 19, 2023 at 02:38:34PM +0000, Niklas Cassel wrote:
> > On Tue, Dec 19, 2023 at 09:20:21AM -0500, Frank Li wrote:
> > > On Tue, Dec 19, 2023 at 10:07:14AM +0000, Niklas Cassel wrote:
> > > > On Mon, Dec 18, 2023 at 11:48:43PM -0500, Frank Li wrote:
> > > > > dw_pcie_ep_inbound_atu()
> > > > > {
> > > > > 	...
> > > > > 	if (!ep->bar_to_atu[bar])
> > > > > 		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
> > > > > 	else
> > > > > 		free_win = ep->bar_to_atu[bar];
> > > > > 	...
> > > > > }
> > > > > 
> > > > > The atu index 0 is valid case for atu number. The find_first_zero_bit()
> > > > > will return 6 when second time call into this function if atu is 0. Suppose
> > > > > it should use branch 'free_win = ep->bar_to_atu[bar]'.
> > > > > 
> > > > > Change 'bar_to_atu' to s8. Initialize bar_to_atu as -1 to indicate it have
> > > > > not allocate atu to the bar.
> > > > > 
> > > > > Reported-by: Niklas Cassel <Niklas.Cassel@wdc.com>
> > > > > Close: https://lore.kernel.org/linux-pci/ZXt2A+Fusfz3luQV@x1-carbon/T/#u
> > > > > Fixes: 4284c88fff0e ("PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address")
> > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > > ---
> > > > > 
> > > > > Notes:
> > > > >     @Niklas:
> > > > >     	I have not test your case. I should be equal to previous's fix in
> > > > >     mail list.
> > > > 
> > > > Hello Frank,
> > > > 
> > > > Thank you for sending a proper fix for this!
> > > > 
> > > > Personally, I slightly prefer your fix that saves the iatu index + 1, and
> > > > keeps 0 to mean unused. That way, you don't need the memset, and you don't
> > > > need to change the type to signed, but either way is fine by me, so:
> > > 
> > > index + 1 don't match hardware iATU index. It will be confused because
> > > other parts is 0 based.
> > > 
> > > So I choose "-1" as free iATU.
> > 
> > A s8 can hold a max value of 127.
> > CX_ATU_NUM_OUTBOUND_REGIONS seems to be 0-255.
> > 
> > Since the DWC code can be synthesized with 256 iATUs,
> > your code will not work on systems with 128 or more iATUs.
> > 
> > If we continue to use a u8, and offset the saved value by one,
> > we will at least be able to support 255-1 == 254 iATUs.
> > 
> 
> Agree. I cannot suggest a better alternative. So let's go with this. But please
> add a comment before bar_to_atu assignment to make it clear. Like,
> 
> 	/*
> 	 * Always increment free_win before assignment, since value 0 is used to
> 	 * identify unallocated mapping.
> 	 */
> 	ep->bar_to_atu[bar] = free_win + 1;

This patch already change to use "-1" as free. Only issue for this patch is
that use 's16' instead of 's8' becasue max ATU number is 255.

Frank

> 
> - Mani
> 
> > 
> > Kind regards,
> > Niklas
> 
> -- 
> மணிவண்ணன் சதாசிவம்