Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Subject: Re: [PATCH v2 4/5] PCI: keystone: Add quirk to mark AM654 RC BAR flag
 as IORESOURCE_UNSET
From:   Kishon Vijay Abraham I <kishon@ti.com>
To:     Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
        Bjorn Helgaas <bhelgaas@google.com>
CC:     Rob Herring <robh@kernel.org>,
        =?UTF-8?Q?Krzysztof_Wilczy=c5=84ski?= <kw@linux.com>,
        <linux-pci@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <devicetree@vger.kernel.org>
References: <20211126083119.16570-1-kishon@ti.com>
 <20211126083119.16570-5-kishon@ti.com> <20220104155741.GA28358@lpieralisi>
 <f3a2c3f0-caf2-743c-a2f7-a99ea3ddb04f@ti.com>
Message-ID: <a07040ce-e043-22ac-2ee5-47a3bfdedd3b@ti.com>
Date:   Fri, 4 Feb 2022 20:38:46 +0530
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <f3a2c3f0-caf2-743c-a2f7-a99ea3ddb04f@ti.com>
Content-Type: text/plain; charset="utf-8"
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi Lorenzo,

On 11/01/22 11:53 am, Kishon Vijay Abraham I wrote:
> Hi Lorenzo,
> 
> On 04/01/22 9:27 pm, Lorenzo Pieralisi wrote:
>> On Fri, Nov 26, 2021 at 02:01:18PM +0530, Kishon Vijay Abraham I wrote:
>>> AM654 RootComplex has a hard coded 64 bit BAR of size 1MB and also has
>>> both MSI and MSI-X capability in it's config space. If PCIEPORTBUS is
>>> enabled, it tries to configure MSI-X and msix_mask_all() adds about 10
>>> Second boot up delay when it tries to write to undefined location.
>>>
>>> Add quirk to mark AM654 RC BAR flag as IORESOURCE_UNSET so that
>>> msix_map_region() returns NULL for Root Complex and avoid un-desirable
>>> writes to MSI-X table.
>>
>> I don't think this is the right fix (it is not even a fix, just a
>> plaster to workaround an issue).
>>
>> What do you mean by "writing to an undefined location" ?
>>
>> What does "a hard coded BAR" mean ?
>>
>> What happens if we _rightly_ write into it (ie to size it) ?
> 
> There are two parts w.r.t setting the BAR; one is during the configuration and
> the other is during the enumeration.
> i) During the configuration, the size of the BAR is configured and the inbound
> ATU is configured to map the BAR to a physical memory.
> ii) During the enumeration, the size of the BAR is obtained and an address is
> allocated and programmed in the BAR.
> 
> In the case of RC, for (i) above, the BAR size is configured as '0'
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-designware-host.c#n556
> and the inbound ATU is not programmed at all.
> 
> However, in the case of AM654, the HW configures BAR0 for a fixed size of 1MB
> (irrespective of what SW programmed in [i]). While this was done more for a
> endpoint usecase, since the same IP is configured for both RC mode and EP mode,
> the fixed BAR size is seen with RC mode as well. AM654 also has MSI-X capability
> for RC mode (the IP should have been ideally configured to have MSI-X capability
> for EP mode). This results in PCIEPORTBUS doing some undesired access in
> msix_mask_all().
> 
> Here I configure IORESOURCE_UNSET so that memory is not allocated for RC BAR.

Do you need further clarifications on this?

Thank You,
Kishon

> 
> Thank You,
> Kishon
> 
> 
>>
>> Lorenzo
>>
>>> Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com>
>>> ---
>>>  drivers/pci/controller/dwc/pci-keystone.c | 8 +++++++-
>>>  1 file changed, 7 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/pci/controller/dwc/pci-keystone.c b/drivers/pci/controller/dwc/pci-keystone.c
>>> index 52d20fe17ee9..73e6626a0d8f 100644
>>> --- a/drivers/pci/controller/dwc/pci-keystone.c
>>> +++ b/drivers/pci/controller/dwc/pci-keystone.c
>>> @@ -557,8 +557,14 @@ static void ks_pcie_quirk(struct pci_dev *dev)
>>>  		{ 0, },
>>>  	};
>>>  
>>> -	if (pci_is_root_bus(bus))
>>> +	if (pci_is_root_bus(bus)) {
>>>  		bridge = dev;
>>> +		if (pci_match_id(am6_pci_devids, bridge)) {
>>> +			struct resource *r = &dev->resource[0];
>>> +
>>> +			r->flags |= IORESOURCE_UNSET;
>>> +		}
>>> +	}
>>>  
>>>  	/* look for the host bridge */
>>>  	while (!pci_is_root_bus(bus)) {
>>> -- 
>>> 2.17.1
>>>