Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38;
Date:   Wed, 18 Oct 2023 13:42:02 +0300
From:   Serge Semin <fancer.lancer@gmail.com>
To:     Siddharth Vadapalli <s-vadapalli@ti.com>
Cc:     Bjorn Helgaas <helgaas@kernel.org>, lpieralisi@kernel.org,
        robh@kernel.org, kw@linux.com, bhelgaas@google.com,
        linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org, r-gunasekaran@ti.com,
        srk@ti.com
Subject: Re: [PATCH] PCI: keystone: Don't enable BAR0 if link is not detected
Message-ID: <tlx5ysa4jnqzmhq5dpk5yk6uuddpwl5h2ieg6n2h3ozksqyx44@yx5mhvwg5miv>
References: <20231013184958.GA1118393@bhelgaas>
 <c11f4b9f-8cbe-1fd0-886b-f36547dc8d3c@ti.com>
 <klxzte53bzk774zinhfrdwdwalvv2hlvc2mqiuyecxcneqkdbt@qbkyc4fdlcka>
 <eacc8aa9-7480-f46c-8852-88f1f8f46bff@ti.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <eacc8aa9-7480-f46c-8852-88f1f8f46bff@ti.com>
Precedence: bulk

On Tue, Oct 17, 2023 at 09:44:51AM +0530, Siddharth Vadapalli wrote:
> Hello,
> 
> On 17/10/23 02:59, Serge Semin wrote:
> > Hi Siddharth
> > 
> 
> ...
> 
> >>>
> >>> I assume MSI-X actually does work for downstream endpoints?  I
> >>> wouldn't think anybody would have bothered with
> >>> ks_pcie_v3_65_add_bus() unless MSI-X works.
> >>
> >> Yes, I think it is supposed to work, but it doesn't seem to be working right now
> >> considering that even with Endpoint device connected, the readl() returns all Fs.
> > 
> > Could you please have look at what DW PCIe IP-core version is utilized
> > in the Keystone PCIe host controller? If it's of v5.x then here is
> 

> The DW PCIe IP-core version is 4.90a.
> 
> > what HW databook says about the BARs initialization: "If you do use a
> > BAR, then you should program it to capture TLPs that are targeted to
> > your local non-application memory space residing on TRGT1, and not for
> > the application on TRGT0 (dbi). The BAR range must be outside of the
> > three Base/Limit regions."
> 
> Yes, it's the same even in the DW PCIe IP-core version 4.90a Databook:
> 
> 3.5.7.2 RC Mode
> 
> Two BARs are present but are not expected to be used. You should disable them or
> else they will be unnecessarily assigned memory during device enumeration. If
> you do use a BAR, then you should program it to capture TLPs that are targeted
> to your local non-application memory space residing on TRGT1, and not for the
> application on TRGT1. The BAR range must be outside of the three Base/Limit regions.

Are you really sure that it's 4.90a? Here is what my DW PCIe RC
_v4.90_ HW databook says about the BARs:

"Base Address Registers (Offset: 0x10-x14) The Synopsys core does not
implement the optional BARs for the RC product. This is based on the
assumption that the RC host probably has registers on some other
internal bus and has knowledge and setup access to these registers
already."

What you cited resides in the _v5.x_ databooks. It makes my thinking
that in your case the IP-core isn't of 4.90a version.

-Serge(y)

> 
> > 
> > I have no idea whether the BAR being set with an address within the
> > Base/Limit regions could have caused the lags you see, but I would
> > have at least checked that.
> 
> I will check that. Thank you for sharing your feedback.
> 
> > 
> > -Serge(y)
> > 
> >>
> >> -- 
> >> Regards,
> >> Siddharth.
> 
> -- 
> Regards,
> Siddharth.