Received-SPF: pass (google.com: domain of linux-kernel+bounces-127398-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Tue, 2 Apr 2024 10:42:28 +0530
From: Manivannan Sadhasivam <mani@kernel.org>
To: Niklas Cassel <cassel@kernel.org>
Cc: Bjorn Helgaas <helgaas@kernel.org>, Frank Li <Frank.Li@nxp.com>,
	niklas.cassel@wdc.com, bhelgaas@google.com,
	gustavo.pimentel@synopsys.com, imx@lists.linux.dev,
	jdmason@kudzu.us, jingoohan1@gmail.com, kw@linux.com,
	linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org,
	lpieralisi@kernel.org, robh@kernel.org
Subject: Re: [PATCH v2 1/1] PCI: dwc: Fix index 0 incorrectly being
 interpreted as a free ATU slot
Message-ID: <20240402051228.GH2933@thinkpad>
References: <20240321171345.GA2385@thinkpad>
 <20240321180732.GA1329092@bhelgaas>
 <20240322052623.GA4092@thinkpad>
 <Zf0i1X5fg-E59NWx@ryzen>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <Zf0i1X5fg-E59NWx@ryzen>

On Fri, Mar 22, 2024 at 07:19:01AM +0100, Niklas Cassel wrote:
> On Fri, Mar 22, 2024 at 10:56:23AM +0530, Manivannan Sadhasivam wrote:
> > On Thu, Mar 21, 2024 at 01:07:32PM -0500, Bjorn Helgaas wrote:
> > > On Thu, Mar 21, 2024 at 10:43:45PM +0530, Manivannan Sadhasivam wrote:
> > > > On Mon, Mar 04, 2024 at 05:46:16PM -0500, Frank Li wrote:
> > > > > dw_pcie_ep_inbound_atu()
> > > > > {
> > > > > 	...
> > > > > 	if (!ep->bar_to_atu[bar])
> > > > > 		free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows);
> > > > > 	else
> > > > > 		free_win = ep->bar_to_atu[bar];
> > > > > 	...
> > > > > }
> > > > > 
> > > > > The atu index 0 is valid case for atu number. The find_first_zero_bit()
> > > > > will return 6 when second time call into this function if atu is 0. Suppose
> > > > > it should use branch 'free_win = ep->bar_to_atu[bar]'.
> > > > > 
> > > > > Change 'bar_to_atu' to free_win + 1. Initialize bar_to_atu as 0 to indicate
> > > > > it have not allocate atu to the bar.
> > > > 
> > > > I'd rewrite the commit message as below:
> > > > 
> > > > "The mapping between PCI BAR and iATU inbound window are maintained in the
> > > > dw_pcie_ep::bar_to_atu[] array. While allocating a new inbound iATU map for a
> > > > BAR, dw_pcie_ep_inbound_atu() API will first check for the availability of the
> > > > existing mapping in the array and if it is not found (i.e., value in the array
> > > > indexed by the BAR is found to be 0), then it will allocate a new map value
> > > > using find_first_zero_bit().
> > > > 
> > > > The issue here is, the existing logic failed to consider the fact that the map
> > > > value '0' is a valid value for BAR0. Because, find_first_zero_bit() will return
> > > > '0' as the map value for BAR0 (note that it returns the first zero bit
> > > > position).
> > > > 
> > > > Due to this, when PERST# assert + deassert happens on the PERST# supported
> > > > platforms, the inbound window allocation restarts from BAR0 and the existing
> > > > logic to find the BAR mapping will return '6' for BAR0 instead of '0' due to the
> > > > fact that it considers '0' as an invalid map value.
> > > > 
> > > > So fix this issue by always incrementing the map value before assigning to
> > > > bar_to_atu[] array and then decrementing it while fetching. This will make sure
> > > > that the map value '0' always represents the invalid mapping."
> > > 
> > > This translates C code to English in great detail, but still doesn't
> > > tell me what's broken from a user's point of view, how urgent the fix
> > > is, or how it should be handled.
> > > 
> > > DMA doesn't work because ATU setup is wrong?  Driver MMIO access to
> > > the device doesn't work?  OS crashes?  How?  Incorrectly routed access
> > > causes UR response?  Happens on every boot?  Only after a reboot or
> > > controller reset?  What platforms are affected?  "PERST# supported
> > > platforms" is not actionable without a lot of research or pre-existing
> > > knowledge.  Should this be backported to -stable?
> > > 
> > 
> > Severity is less for the bug fixed by this patch. We have 8 inbound iATU windows
> > on almost all of the platforms and after PERST# assert + deassert, BAR0 uses map
> > '6' instead of '0'.
> > 
> > This has no user visibility since the mapping will go fine and we have only 6
> > BARs. So I'd not mark this as as critical fix that needs special attention.
> 
> So we will have 6 mappings configured, but only 5 BARs, so two mappings for
> BAR0. The iATU looks at them in order, so index 0 will override index 6.
> 
> We are lucky that the endpoint subsystem does not clean up allocations properly
> right now (you have an outstanding series which fixes this).
> 
> If the endpoint subsystem did clean up resources properly, we would DMA to the
> area that was previously allocated for BAR0, instead of the new area for BAR0.
> 

How would DMA happen to the previously allocated area? When the BARs are cleared
properly and then allocated again, BAR0 will get the map of 0 again and then the
iATU will map window 0 with BAR0. Only if we don't clear the BARs properly (as
like now), then it will result in BAR0 having 2 identical mappings and even with
that there won't be any issue since both map 0 and 6 are valid.

Am I missing anything?

> Perhaps just include this fix in your series that actually cleans up the BARs?
> 

Yeah, makes sense. Once we agree on a finalized commit message in this thread,
I'll include this patch in my series.

- Mani

-- 
மணிவண்ணன் சதாசிவம்