Received-SPF: pass (google.com: domain of linux-kernel+bounces-66783-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <17dadf66-055c-4796-a905-44e37b6fcfe3@nvidia.com>
Date: Thu, 15 Feb 2024 16:55:47 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH V16 13/13] PCI: tegra: Add Tegra194 PCIe support
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: Lorenzo Pieralisi <lpieralisi@kernel.org>, bhelgaas@google.com,
 robh+dt@kernel.org, mark.rutland@arm.com, thierry.reding@gmail.com,
 jonathanh@nvidia.com, Kishon Vijay Abraham I <kishon@kernel.org>,
 catalin.marinas@arm.com, Will Deacon <will@kernel.org>,
 jingoohan1@gmail.com, gustavo.pimentel@synopsys.com, digetx@gmail.com,
 mperttunen@nvidia.com, linux-pci@vger.kernel.org,
 devicetree@vger.kernel.org, linux-tegra@vger.kernel.org,
 linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org,
 kthota@nvidia.com, mmaddireddy@nvidia.com, sagar.tv@gmail.com
References: <20240214191249.GA1263222@bhelgaas>
Content-Language: en-US
From: Vidya Sagar <vidyas@nvidia.com>
In-Reply-To: <20240214191249.GA1263222@bhelgaas>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?MHNuTW1TOGpqRzFyU3Q5TXY5VE1kTGd6RDdIZ2FLblE2VkJidEZHejBkYTZG?=
 =?utf-8?B?cFkzZnlRbWYxZDVYSSs1Q2tkZENFb0g3WExUb1c0Z3RzQ1VvMWRjeCtKdTVi?=
 =?utf-8?B?SDNVNDh1aktrd2hsWHl3Q1RTOWxXN29qb1haRTBKQ05RUmFLSlU5R2pCcENm?=
 =?utf-8?B?b0F4Y2NKWGFOUWVJTWpkTTNBRFVYRzNLV2NleHNmQW93TjZ6L0xHM3ZPRm1J?=
 =?utf-8?B?ZDl1RDdBbHJ3dU1NQll5amd3T2hOMEh5NGRISHJYSDV3NWtMYlZLUjdEMUs0?=
 =?utf-8?B?N3JFRnlPL0YyYk8zemM3NCtOT3laQ1NjQ0V5WmZsdjRyaXUvYlBPSlJWQVVk?=
 =?utf-8?B?WjhaZkRvUVMxTXFJZ3ZWamVGZStEV1NVZGVtRllqZUhhb2VlWnJxRHhZSTVq?=
 =?utf-8?B?emRRZ0tNRmQwZElhOU0vcmpxV05NaFZwUnZHZXpscXUyS2RwMjJ6ZmNSSjZw?=
 =?utf-8?B?Ny9ZTHhJZzRBZU1TZ3Bab2RBZ2RxVjlpM2djdWFWa21TTlJIRmN5bTFMdEcw?=
 =?utf-8?B?YjBIelVRQjVxa0VQZElLdWVFeERyOHpEQm13UUs2a2IyQkRJUGluQThDMm5i?=
 =?utf-8?B?eXhTbWdMdFVuWjRaWWZDR05mMFcyZisyQmNwT2MvdFI0alpvWFA2RlBWZXpo?=
 =?utf-8?B?djdSTWVjNXdhT1hyL3NXY2RtUnY1WWswaW9YcXp3MkxpOXMwSHRPbE5KU0x2?=
 =?utf-8?B?UWhOMTJNWHNXTkhBNHo4SUJITWxCK3l5cnZRd2hsNUVubk40dVJCSjhwL2g4?=
 =?utf-8?B?TkRHQ25ZcHk2RFk3ZEdYOHk2VGs0ZzI3ZEU2b2l2S1Fwd2JtbzRLV21QZ0lI?=
 =?utf-8?B?K09xY0RmS2srT3h0T0ZySldVNC9FSTdRQStiRVRzRzROUVhtOU9RWkFIek93?=
 =?utf-8?B?Sk9VY1BuWml3Zk1tS1ByakRXVENYUnhBSnk1eTFGQ3FKQWE0czU5cDdoS3Z6?=
 =?utf-8?B?OUUxVEFpUVVmS1hkbHRUM3U5d2lBb2JNdHZ2T1hYNDdCMy93UUY5U0JLUE1h?=
 =?utf-8?B?cFBGd2s1aG9mZWo1OVp2bU8zZlRWcGpkSDZabDNVV3dzSWpmZThkZExJL2gv?=
 =?utf-8?B?aitWTWVHNks4TFdOdGlyL0dkL29IMU9ZanozNitORHdkNVdGbW84UVo1ay9q?=
 =?utf-8?B?cjZvU3c0eHloRzhuUkI5WkY4NnZzU29FbUN2L0gvTDRSL3IwaVZpN0hhdDdK?=
 =?utf-8?B?b2puVWlFa1l6R01Wc1dZL3llYzZ6VFlWL3c5cVNoOFE1WnBOeGpKRmJ3MldR?=
 =?utf-8?B?aDZkbjB6a2V5SGUzL1lVYXVvYllpdW1GT0FyRHJUbkhzNVdTWm8vT3A4U0RR?=
 =?utf-8?B?WW8wSHp1MTAvYmtJWmttSG9rM1plQjQ2bjNTMk1KZjRLRnZvK29meWpJTGZh?=
 =?utf-8?B?VXRINE14bkhFWGRoWmxQdGtSWndzRk1ZcGw2MVM2Qk1BblU4NFpTNWVJU3dB?=
 =?utf-8?B?VVNyT0hDdm85WGV3NmpPd2hJeHMyMkJZWG43NVk4OFhldS9nSHhENFVqWjV3?=
 =?utf-8?B?bnhLRmlmZE02Ky83Q04rREhkOG82SHMzUXAyYW5TQWxwL3h1TDJQT1lHY09S?=
 =?utf-8?B?bDFmUFF1aDI1S0lUQlIzaGMrS0gvcWlyRSs2d2V2dUp3Z3JjTWR6UDJOTHZW?=
 =?utf-8?B?L1FjUkZWSVRHdkxjMDQ4b3hxTzhOWE9ER2piOVpWTDhxK0N4Tk5PaFhEODJx?=
 =?utf-8?B?Qm9wdnoyblZaQzdxVlNNUDVsM2hrUjJoa2ZWMmEzeGpOblVhQ2VXdzFGakxF?=
 =?utf-8?B?R3BBYjVQMy9QVmN5SWtRWE81d09TMGU5blhrUzNOOE4xMUlwWi94YXYrenJl?=
 =?utf-8?B?dGtpczMvVmdTU3lmRmNpNlFUaEQ1bHJiSDJPOFpsN01QNk5QT3JiaDBJNXF2?=
 =?utf-8?B?L3ZpQjNNeHYwYml1YUdTbnNXWDArb3NoVFFTeXFjUDlUanhRcjVGbUkwVlhU?=
 =?utf-8?B?Ym9PSm13ZEx5UXF3bVVXS1hsaGJKa1VWcE1Dd1JnL3VlWGdSam1VTGIwMllI?=
 =?utf-8?B?NG03Umg0K2l4bUJ5WlBrYVpvd0lZMThTTXRrZzZPclJwWnEvSXFqZEthdjd0?=
 =?utf-8?B?SVNtanA5eVAxb3YyeTduUFNtbE04TjdCYWZla3FyWVpLcVo4a0RMbERnTHVm?=
 =?utf-8?Q?ZzoIFDAcoMaSr8wjg3QswyqUp?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 394e01f4-36e1-4c7f-b780-08dc2e18e201
X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB6674.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2024 11:25:58.9251
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: urtg9EGQwTj1iO0BEOirQTQg2ogPTIdVguHBs7L+JNLwfuTF9fwC7zxB9ALxrzYS2L3jvxpWoACxyx94kJoRiQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6661


On 15-02-2024 00:42, Bjorn Helgaas wrote:
> External email: Use caution opening links or attachments
>
>
> Hi Vidya, question about ancient history:
>
> On Tue, Aug 13, 2019 at 05:06:27PM +0530, Vidya Sagar wrote:
>> Add support for Synopsys DesignWare core IP based PCIe host controller
>> present in Tegra194 SoC.
>> ...
>> +static int tegra_pcie_dw_host_init(struct pcie_port *pp)
>> +{
>> +     struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>> +     struct tegra_pcie_dw *pcie = to_tegra_pcie(pci);
>> +     u32 val, tmp, offset, speed;
>> +
>> +     tegra_pcie_prepare_host(pp);
>> +
>> +     if (dw_pcie_wait_for_link(pci)) {
>> +             /*
>> +              * There are some endpoints which can't get the link up if
>> +              * root port has Data Link Feature (DLF) enabled.
>> +              * Refer Spec rev 4.0 ver 1.0 sec 3.4.2 & 7.7.4 for more info
>> +              * on Scaled Flow Control and DLF.
>> +              * So, need to confirm that is indeed the case here and attempt
>> +              * link up once again with DLF disabled.
> This comment suggests that there's an issue with *Endpoints*, not an
> issue with the Root Port.  If so, it seems like this problem could
> occur with all Root Ports, not just Tegra194.  Do you remember any
> details about this?
>
> I don't remember hearing about any similar issues, and this driver is
> the only place PCI_EXT_CAP_ID_DLF is referenced, so maybe it is
> actually something related to Tegra194?
We noticed PCIe link-up issues with some endpoints. link-up at the physical
layer level but NOT at the Data link layer level precisely. We further
figured out that it is the DLFE DLLPs that the root port sends during the
link up process which are causing the endpoints get confused and preventing
them from sending the InitFC DLLPs leading to the link not being up at
Data Link Layer level.

PCIe spec rev4.0, Rev 4.0, ver 1.0, Section-3.3 says that (verbatim) 
"The Data
Link Feature Exchange Enable field permits systems to disable the Data Link
Feature Exchange. This can be used to work around legacy hardware that does
not correctly ignore the DLLP".

Based on this, we added the following code to retry link-up with DLFE 
disabled,
only upon confirming that the link came up at phy level but not at the 
DL level
in the previous attempt.

Thanks,

Vidya Sagar

>
>> +             val = appl_readl(pcie, APPL_DEBUG);
>> +             val &= APPL_DEBUG_LTSSM_STATE_MASK;
>> +             val >>= APPL_DEBUG_LTSSM_STATE_SHIFT;
>> +             tmp = appl_readl(pcie, APPL_LINK_STATUS);
>> +             tmp &= APPL_LINK_STATUS_RDLH_LINK_UP;
>> +             if (!(val == 0x11 && !tmp)) {
>> +                     /* Link is down for all good reasons */
>> +                     return 0;
>> +             }
>> +
>> +             dev_info(pci->dev, "Link is down in DLL");
>> +             dev_info(pci->dev, "Trying again with DLFE disabled\n");
>> +             /* Disable LTSSM */
>> +             val = appl_readl(pcie, APPL_CTRL);
>> +             val &= ~APPL_CTRL_LTSSM_EN;
>> +             appl_writel(pcie, val, APPL_CTRL);
>> +
>> +             reset_control_assert(pcie->core_rst);
>> +             reset_control_deassert(pcie->core_rst);
>> +
>> +             offset = dw_pcie_find_ext_capability(pci, PCI_EXT_CAP_ID_DLF);
>> +             val = dw_pcie_readl_dbi(pci, offset + PCI_DLF_CAP);
>> +             val &= ~PCI_DLF_EXCHANGE_ENABLE;
>> +             dw_pcie_writel_dbi(pci, offset, val);
>> +
>> +             tegra_pcie_prepare_host(pp);
>> +
>> +             if (dw_pcie_wait_for_link(pci))
>> +                     return 0;
>> +     }
>> +
>> +     speed = dw_pcie_readw_dbi(pci, pcie->pcie_cap_base + PCI_EXP_LNKSTA) &
>> +             PCI_EXP_LNKSTA_CLS;
>> +     clk_set_rate(pcie->core_clk, pcie_gen_freq[speed - 1]);
>> +
>> +     tegra_pcie_enable_interrupts(pp);
>> +
>> +     return 0;
>> +}