Received-SPF: pass (google.com: domain of linux-kernel+bounces-4944-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <0fd5c09f-1cf2-8813-a8f9-1bd856e3a298@amd.com>
Date: Tue, 19 Dec 2023 14:41:08 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.1
Subject: Re: [Patch v2 2/2] thunderbolt: Teardown tunnels and reset downstream
 ports created by boot firmware
Content-Language: en-US
To: Mika Westerberg <mika.westerberg@linux.intel.com>
Cc: Sanath S <Sanath.S@amd.com>, mario.limonciello@amd.com,
 andreas.noever@gmail.com, michael.jamet@intel.com, YehezkelShB@gmail.com,
 linux-usb@vger.kernel.org, linux-kernel@vger.kernel.org
References: <ff143967-63ff-c4fb-9c88-8537a663c45b@amd.com>
 <20231215115521.GW1074920@black.fi.intel.com>
 <c7d174d3-028f-9ce4-7ef5-3e033c195159@amd.com>
 <20231215140224.GX1074920@black.fi.intel.com>
 <866cb714-b9a8-a7d4-4c59-6ba771ef325f@amd.com>
 <20231218104234.GB1074920@black.fi.intel.com>
 <c433f29b-597c-b6d6-aa48-2b84a26dc623@amd.com>
 <20231218113151.GC1074920@black.fi.intel.com>
 <20231218122312.GE1074920@black.fi.intel.com>
 <997f2a94-66d9-fb95-8f75-46d61937f7e8@amd.com>
 <20231218131840.GH1074920@black.fi.intel.com>
From: Sanath S <sanaths2@amd.com>
In-Reply-To: <20231218131840.GH1074920@black.fi.intel.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?ZkI1M1pVaWtQTFRyaXBjQnNqUHNUaHZuc3hxVW1MMmppdGRDNGpIaW02Rmhk?=
 =?utf-8?B?Qko0RmpWSS9tY0I4ZmZ0RUtuUWlrSFhSc1lkUGN1Y0ptSG1pLzVNR3AyTi9h?=
 =?utf-8?B?TUwwQlZZVjJyTDFJYmZ3T3BkcElMb0VGR3N3cFdRRXZxdmg4azFJcE1iUEE4?=
 =?utf-8?B?OVJSQWJSWVZlRkZUQ1NoclVXRnYvcm5JZVlNcUZyK2psdDZ6SE1PRnEvV3Vp?=
 =?utf-8?B?cmU0U0haQ0lFQnlpQlMwczhQN1F4TmloVFdCaDljS2NEd2N1bWRDRXZZOEs4?=
 =?utf-8?B?cHg2VStIazBhM00xN1VvL1BzbmgzdTNBREV5clhRTlgyZVB5My9WTGNUdW5U?=
 =?utf-8?B?ejI2NWpkYjJuSVp0NmUrOXo0eVV1SlVuYTF1akxEaFExdkd5TWhHWnhDdjNR?=
 =?utf-8?B?c0xEMFcvR1lPYWlVLy96T05HRWdlTm5uNCs5VitmbHFheFFKNm1HRUhiM2Nj?=
 =?utf-8?B?SmszVFVRQ0V5THlqL3Y0OUc5dDM1Z3Z4cHYwczFlay9UbE5aZVFRNVl5Ulh6?=
 =?utf-8?B?MXdUL2tLTmE3b3NZQklLMVNTZnNLSGQzN3JaaHBjNWFkNk1PYUFITnQ3U1NU?=
 =?utf-8?B?aGhrYWEvNHZ0c21HZkdGT1IvdUM5ZGcyZnJXbHF6WlA4QktTZGRXU1Qxd2g1?=
 =?utf-8?B?ZGtadmZ0SWJiRzF2NUcvanVtUFVxSC9uYjJOcmVBMnI4Q1RVUEZsaTFrczNE?=
 =?utf-8?B?RnRaaStFZk5uMXVWMnZoaFc1NFF4S093cDA2SUZwU2NiK0N5dW5teTVjNjkz?=
 =?utf-8?B?cUYxV1pzU3RJN29ianJ6RjJMZWRIcGlqNmkrbXZNN05Vc0FOUmw0c05YTEtS?=
 =?utf-8?B?eUJiM281WGJERWpDckxyTHBySEFVNHVSWmhhd2lTbzJTMVJoQUFyOEJmNWN1?=
 =?utf-8?B?SWpKa01XNHpLQnkrNVAzUWhiUUlvTWtXWFZzNHBSaFRiMFdOQVFKM3NzMEdO?=
 =?utf-8?B?UkgrOXVDRURncEdHZUJZRUFvbmRvMW5NVmp1Uk5EMldXSUhWeE1EdHVMRGZT?=
 =?utf-8?B?SUlFNlV1ejcrQ3cvMHVHUEtwMk1SUnJudXFEY2dCYWJNemQ1b1ViRGY0K3Zw?=
 =?utf-8?B?M3g5MG5yYlhGdTluSy80cWprbE1HVUdrZzgzdlp3eEp1TTl4TjYvcnIyNThz?=
 =?utf-8?B?Z2FJSTNUZ0QwZmRzV29jVHhHMEtXRDVrNDZyb2ZjYWorbnlnYm56MWJwSzNa?=
 =?utf-8?B?WXZQYWsxVm5JdFliT2xITHpoa1Riamg2dmcxTktUR1JySnB4ZzFxOEpOTTA0?=
 =?utf-8?B?RG93bmpySVcvQkZoUkk4dEMyRjlocDVMMjZrOTduUkk4RGl2MTNiSVN3SDJk?=
 =?utf-8?B?SlU5QUVpd1pvSjA5SEJ0QTg0TDBBaGdabVJiK2ZESUhOd1FWQkVHWG50OWJj?=
 =?utf-8?B?VnRiUlUvV1RSRG10aTdSWGhQeUNrYm9SOWNjSkQ1RHNYRjloMElwdDFtcGEy?=
 =?utf-8?B?YjVHUVdxSGQ4UWtVd296aTdGc2NQRjEzWEwwRWNTZElXOC8rdDZFQ1dFZjEv?=
 =?utf-8?B?anJSdFFKWXRsTE1PMGVmclRpUUF0QnRsSnVNaFoyOHdCeUZFWmttK2Q3c2J4?=
 =?utf-8?B?R0xtVzBRRXEvdVdpTzBGMjM4YUpvWnhlQmNJeDhZTm96RTJyZlhCcDNpSXlK?=
 =?utf-8?B?V1o5UmhCMXFzYVdieFg3cFdGRU5mZFB4ZVVBVG9lbUVhTklPeVBUQk9nVHMv?=
 =?utf-8?B?UndCcnN3U0JKUUtsWjdGYTBSQ3p4VUoyWUp4ZzRYQjhERXJpT1MzR3RNT2pw?=
 =?utf-8?B?dHJzNEp2dDNhR2xQdHYrMnFZNkdBTEpuQ3Roa1plSkplME84TXdMZzVoTytq?=
 =?utf-8?B?cDdZeHU1eVNURm53ZDNySStCcUhoMkR6aDZCY1dKTWxpQ1N5V1FNRWM4dHJi?=
 =?utf-8?B?NWtkMVBwWUtWZ1ZIR1B2RFU0MmlSTWtyYnVrWHhvTUxldjFXSHowZGEzd2Vr?=
 =?utf-8?B?a2kwQmp4Mnh2YlluNFZSWW9BTmxoSCtCZUxuZ09Kb0VHTzFnUEdKWWFWRGdl?=
 =?utf-8?B?WWY0Tm9ZQk1EMG45dk02ZWpkYmNFUUhMVFdFMEY5WFNQNVJ0WEYwMU9TZkhq?=
 =?utf-8?B?aEg0cFdpcnRUV3FZUG04ZWZBZkplblBsMjVvdUM5dm04L2JMeHZGWlZXWEht?=
 =?utf-8?Q?Z5gsXuErTFhPMcYzLWZpNk/ju?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f5853a1d-7881-41f5-f810-08dc00727aff
X-MS-Exchange-CrossTenant-AuthSource: MW4PR12MB7016.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 09:11:27.7245
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LzhQmDECxSjjNp9loCiSyGfwgI89X2CWXwP+TwblAitAyoNKIeGkwEAlXieWyEnlmwjsBNWzv32pa/NDM0FLeg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8457


On 12/18/2023 6:48 PM, Mika Westerberg wrote:
> On Mon, Dec 18, 2023 at 06:35:13PM +0530, Sanath S wrote:
>> On 12/18/2023 5:53 PM, Mika Westerberg wrote:
>>> On Mon, Dec 18, 2023 at 01:31:51PM +0200, Mika Westerberg wrote:
>>>> On Mon, Dec 18, 2023 at 04:49:13PM +0530, Sanath S wrote:
>>>>>> The discover part should not do anything (like write the hardware) so
>>>>>> perhaps it is just some timing thing (but that's weird too).
>>>>>>
>>>>>> I think we should do something like this:
>>>>>>
>>>>>> 1. Disable all enabled protocol adapters (reset them to defaults).
>>>>>> 2. Clear all protocol adapter paths.
>>>>>> 3. Issue DPR over all enabled USB4 ports.
>>>>>>
>>>>>> BTW, what you mean "didn't work"?
>>>>> Path activation would go fine after DPR like below:
>>>>>
>>>>> [   15.090905] thunderbolt 0000:c4:00.5: 0:5 <-> 2:9 (PCI): activating
>>>>> [   15.090932] thunderbolt 0000:c4:00.5: activating PCIe Down path from 0:5
>>>>> to 2:9
>>>>> [   15.091602] thunderbolt 0000:c4:00.5: activating PCIe Up path from 2:9 to
>>>>> 0:5
>>>>>
>>>>> But, PCIE enumeration doesn't happen (pcie link up will not happen, will not
>>>>> see below logs)
>>>>> [   15.134223] pcieport 0000:00:03.1: pciehp: Slot(0-1): Card present
>>>>> [   15.134243] pcieport 0000:00:03.1: pciehp: Slot(0-1): Link Up
>>>> Okay, what if you like reset the PCIe adapter config spaces back to the
>>>> defaults? Just as an experiment.
>>> If this turns out to be really complex then I guess it is better to do
>>> it like you did originally using discovery but at least it would be nice
>>> to see what the end result of this experiment looks like :)
I feel it's better to go with discover and then reset for now (as v3).
I'll keep this experiment as "to do" and will send out when I crack it down.
>> Yes, I'll give a try.
>> As an experiment, I tried to compare the path deactivation that occurs at
>> two place.
>> 1. In tb_switch_reset where we are calling tb_path_deactivate_hop(port, i).
>> 2. While we get a unplug event after doing DPR.
>>
>> I observed both have different hop_index and port numbers.
>> So, are we calling tb_path_deactivate_hop with wrong hop ids ?
> Wrong adapters possibly.
>
>>  From deactivate tunnel (called while unplug) :
>> [    3.408268] thunderbolt 0000:c4:00.5: deactivating PCIe Down path from
>> 2:9 to 0:5
>> [    3.408282] deactivate hop port = 9 hop_index=8
>> [    3.408328] deactivate hop port = 2 hop_index=10
> Definitely should be port = 5 (that's PCIe down in your log) and
> hop_index = 8 (that's the one used with PCIe).
>
>> Deactivate from tb_switch_reset() :
>> deactivate hop port = 5 hop_index=8
> Can you add some more logging and provide me the dmesg or
> alternativively investigate it yourself. You can use tb_port_dbg() to
> get the port numbers to the log.
I've sent you complete dmesg.

Here is the log w.r.t port numbers and path clean up.

[    3.389038] thunderbolt 0000:c4:00.5: 0:3: Downstream port, setting DPR
[    3.389065] Calling usb4_port_reset
[    3.389068] thunderbolt 0000:c4:00.5: 0:4: Found USB3 DOWN
[    3.389193] thunderbolt 0000:c4:00.5: 0:4: In reset, cleaning up 
path, port->port = 4 hopid = 8
[    3.389203] thunderbolt 0000:c4:00.5: 0:4: deactivating_hop port = 4 
hop_index=8
[    3.389682] thunderbolt 0000:c4:00.5: 0:5: Found PCI Down
[    3.389811] thunderbolt 0000:c4:00.5: 0:5: In reset, cleaning up 
path, port->port = 5 hopid = 8
[    3.389817] thunderbolt 0000:c4:00.5: 0:5: deactivating_hop port = 5 
hop_index=8
[    3.390296] thunderbolt 0000:c4:00.5: 0:6: Found DP IN
[    3.390555] thunderbolt 0000:c4:00.5: 0:6: In reset, cleaning up 
path, port->port = 6 hopid = 8
[    3.390558] thunderbolt 0000:c4:00.5: 0:6: deactivating_hop port = 6 
hop_index=8
[    3.390686] thunderbolt 0000:c4:00.5: 0:6: In reset, cleaning up 
path, port->port = 6 hopid = 9
[    3.390689] thunderbolt 0000:c4:00.5: 0:6: deactivating_hop port = 6 
hop_index=9
[    3.390816] thunderbolt 0000:c4:00.5: 0:7: Found DP IN
[    3.391077] thunderbolt 0000:c4:00.5: 0:7: In reset, cleaning up 
path, port->port = 7 hopid = 8
[    3.391080] thunderbolt 0000:c4:00.5: 0:7: deactivating_hop port = 7 
hop_index=8
[    3.391213] thunderbolt 0000:c4:00.5: 0:7: In reset, cleaning up 
path, port->port = 7 hopid = 9
[    3.391217] thunderbolt 0000:c4:00.5: 0:7: deactivating_hop port = 7 
hop_index=9
[    3.391342] Reset success
[    3.391391] thunderbolt 0000:c4:00.5: 0:2: switch unplugged
[    3.391434] thunderbolt 0000:c4:00.5: 0:4 <-> 2:16 (USB3): deactivating
[    3.391471] thunderbolt 0000:c4:00.5: deactivating USB3 Down path 
from 0:4 to 2:16
[    3.391477] thunderbolt 0000:c4:00.5: 0:4: deactivating_hop port = 4 
hop_index=8
[    3.391641] thunderbolt 0000:c4:00.5: 2:1: deactivating_hop port = 1 
hop_index=9
[    3.391651] thunderbolt 0000:c4:00.5: deactivating USB3 Up path from 
2:16 to 0:4
[    3.391659] thunderbolt 0000:c4:00.5: 2:16: deactivating_hop port = 
16 hop_index=8
[    3.391664] thunderbolt 0000:c4:00.5: 0:2: deactivating_hop port = 2 
hop_index=9
[    3.391701] thunderbolt 0000:c4:00.6: total paths: 3
[    3.391720] thunderbolt 0000:c4:00.6: IOMMU DMA protection is disabled
[    3.392027] thunderbolt 0000:c4:00.5: 0:5 <-> 2:9 (PCI): deactivating
[    3.392154] thunderbolt 0000:c4:00.5: deactivating PCIe Down path 
from 2:9 to 0:5
[    3.392163] thunderbolt 0000:c4:00.5: 2:9: deactivating_hop port = 9 
hop_index=8
[    3.392170] thunderbolt 0000:c4:00.5: 0:2: deactivating_hop port = 2 
hop_index=10
[    3.392534] thunderbolt 0000:c4:00.5: deactivating PCIe Up path from 
0:5 to 2:9
[    3.392539] thunderbolt 0000:c4:00.5: 0:5: deactivating_hop port = 5 
hop_index=8
[    3.392637] thunderbolt 0000:c4:00.5: 2:1: deactivating_hop port = 1 
hop_index=10
[    3.392799] thunderbolt 0-2: device disconnected

But it seems like we are not cleaning up all the paths ?