Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38;
Message-ID: <32163f49-8387-0754-534f-1764e731f26d@amd.com>
Date:   Thu, 14 Dec 2023 12:50:21 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.1
Subject: Re: [Patch v2 2/2] thunderbolt: Teardown tunnels and reset downstream
 ports created by boot firmware
Content-Language: en-US
To:     Mika Westerberg <mika.westerberg@linux.intel.com>
Cc:     Sanath S <Sanath.S@amd.com>, mario.limonciello@amd.com,
        andreas.noever@gmail.com, michael.jamet@intel.com,
        YehezkelShB@gmail.com, linux-usb@vger.kernel.org,
        linux-kernel@vger.kernel.org
References: <20231212191635.2022520-1-Sanath.S@amd.com>
 <20231212191635.2022520-3-Sanath.S@amd.com>
 <20231213054914.GI1074920@black.fi.intel.com>
 <20231213061805.GK1074920@black.fi.intel.com>
 <20231213062306.GL1074920@black.fi.intel.com>
 <adcc6446-8c30-a258-e19b-76fca2c50d21@amd.com>
 <20231213115256.GM1074920@black.fi.intel.com>
 <f673ffc8-f6f8-4898-d809-effb2c24e53e@amd.com>
 <20231214070746.GS1074920@black.fi.intel.com>
From:   Sanath S <sanaths2@amd.com>
In-Reply-To: <20231214070746.GS1074920@black.fi.intel.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RHl4SngzUG5BQXM5WndhSC9kWHRvMlB0RzdOdVUwYTRhMURDTFNDT0Fldkc0?=
 =?utf-8?B?QmY1VTJxdnliVm10ODlDWGNTLzluR0JFT2U2NHhzSUVxcFVXaDhlRWgrcWNF?=
 =?utf-8?B?NlErbXdpNlhCcjRYSTZjdElZaWFqa1RtNldzWTdaSUhjcXdubWtyRFloSW9v?=
 =?utf-8?B?NXZWOUNoaFVrWHpLOWd1VEVySVcrUzd1bUJWaTM0VlJVVkw5RmpSQjNGVWZx?=
 =?utf-8?B?LzhJSitNd0hPNmNmdnhHM09iK2w1THZ3eWN0UG9oUkR5S00xMzZXR3hSZkxD?=
 =?utf-8?B?QXFZbzNJUkxSdE1oRklUMmh6RHJKZ0QxV3dkZWZKNEVqNUFPemFHdXRFOVBP?=
 =?utf-8?B?Rkhpa2hPMEJ3VWJpMGRTSGc5VithaWRpdjBINExyYUxjTHZ0dHVBNE5FUEVC?=
 =?utf-8?B?bTJzODJFdnRYeGIvdWlqQ1NqVzgwMURabTcrQi9jL2c3VWltUnRWRmFYQWU2?=
 =?utf-8?B?ZmNvUWU1VTdydnI0OE1vSlFsaXNRZUpvbENLeDZydzY2UHoydlNEUUs3K2R3?=
 =?utf-8?B?V0Q0ZjI5Z1d3aWg1Y1JYSFB4aElxejhkRHRVeklhU1JmYUZxNGUzNDFIcWZ5?=
 =?utf-8?B?QkJiTlg4NHlzTTJ4c1BTY1Q5SkxmRkl4OC9wOXY3MnZHWm9Lb2xJNWMzcTdh?=
 =?utf-8?B?WGhlVWxKSjdLOXlJYUpNV0IwOFVoUnJTbWJGU25SMmdCeWZCampIUFdJTURS?=
 =?utf-8?B?WjZxY0lweGhxYmJ4OXI1Qys1TDg5M1FzN01tUThnSElYOTBtVjhZUmhOTnVZ?=
 =?utf-8?B?UVg0UUNpY1V3MlBKYTJFRklxTkxWcDNnZnpFQ1BYbHNTdndqSHFJNjd5a2d6?=
 =?utf-8?B?YzZyRU50dnJZVzEyVk5WMEtDWktwY2dGV1l0aXY1T21EMWFzRmxwUEhjSkU3?=
 =?utf-8?B?dlVxaW9DcTNINWZSTC9UOUJiVm90N3ZiUGxudUh6RjNmaG5pVStWaHEybjNG?=
 =?utf-8?B?NDZ3TlpZcnhNYStsZEdYdVAvR0Zlc2pFVGtKRkZHMW5JRkx2MUUyOVFZTUxo?=
 =?utf-8?B?M1FMeHp4OFpjaG8wRmNIREhVQnkxcmkzK1NmSUVEeGZVYzEwdWtpeDVmM3Jk?=
 =?utf-8?B?bDJWSEJHSlFiQk1tbG5GdWhmV1hSQjVXYWp3QmtkUnk1cWN0QTNJUUllMWxS?=
 =?utf-8?B?WldYMktBaHI0L0FGcVdRa2E3cXRyZlZKM052K2t1Wk5ndlU0VnJFS1Z4VzFh?=
 =?utf-8?B?dFVvZUpFRHovSml2OEw3d2gwWnFlWE85b3paK3c2Z1JYVklzdVZUWUJwckUz?=
 =?utf-8?B?U3FyVHhES3c3dGpsTHRKOFQ4TDV0QjczTzFpS1lrVDRyKy83SUZhOTI3Sk5Z?=
 =?utf-8?B?N01UUHlSREN0U2hKUUpDSXBTZGFybnlEVmI0NW1TTU52K241UTFDNytRRUxP?=
 =?utf-8?B?cE1LYm02elZBU0ZURkIwdjliWGtNVmovdEZNamFMbHNuNUhJbExYV1JlbVd0?=
 =?utf-8?B?T1RwUWJPYVIvTFowNFcrL1ZJNHVQS2dNamxSYlZYMHBLbU1XU2tBMTZ1L0ZP?=
 =?utf-8?B?dEVvY3hBcXZ0QzhSc0t3N01Tc2YxajhPcTFnTVlLejFNRDcyZ2kyTUxHMzI4?=
 =?utf-8?B?QjV4c3QrdTVvdjVKcWhoZkdaVVJodGQyWUxSMGtTWjlsbEZNTnlZZHhZWDVR?=
 =?utf-8?B?T0ltY3B5OE9HU0ZPL0Vsb1NaWFY5RzBmcnhYbTVCOFNET2JnRVVvS3JyeHdy?=
 =?utf-8?B?d0dnd2RZdHZIbExwQ0pnZThQMzhyeThHZDMvdUlmNzAyd0VmS2FrektmVnhy?=
 =?utf-8?B?YW9GaUVoY0ljcCtiQkJiQWpkNW5xR3Vrb0FwazVMaE1SeHR0NEJtYm53S3kv?=
 =?utf-8?B?N1RJdmdueWRLU1FQNEZRY0s0d3lEUGVrMUtxdUszdWJMbTMzZjljNVExdWpO?=
 =?utf-8?B?aTlJOHhvOS81MlBQVGJvbDROcEM3SVZSSU5BeFl0RWJSc25mZ0ZFbkF3b0tN?=
 =?utf-8?B?c3FIK2xUdG9oOTJERTB4U2xIbi9CUEY4RFZldFk0NUowbzNaZzUyK2xIU05l?=
 =?utf-8?B?NlJyZk91RWxTU3VXVzdVVTZVMjZBVXFTOUFuUzZQclhRUzQ5anJmSzZtQ0JO?=
 =?utf-8?B?WGladWtWaTlhTWVPcndXVGU4cUVWWDRSem1DWEo3OW8rR2wwVWk2emM0bmF3?=
 =?utf-8?Q?hFfl7p7u6FV9fVzj/XcQMHOys?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b9a9bc99-df2a-4ae3-63d7-08dbfc752792
X-MS-Exchange-CrossTenant-AuthSource: MW4PR12MB7016.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Dec 2023 07:20:31.1595
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: qlpcTTNEw8ncMTmNU8mSG4p3hyWV84g5WMqm7bo9CiV2gbmetHSE9veHYzP3D4+nXPZqQJ7uTpOmBFW4copIsw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5286
X-Spam-Status: No, score=-1.6 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Wed, 13 Dec 2023 23:20:59 -0800 (PST)


On 12/14/2023 12:37 PM, Mika Westerberg wrote:
> On Thu, Dec 14, 2023 at 12:08:34PM +0530, Sanath S wrote:
>> On 12/13/2023 5:22 PM, Mika Westerberg wrote:
>>> On Wed, Dec 13, 2023 at 04:04:57PM +0530, Sanath S wrote:
>>>> On 12/13/2023 11:53 AM, Mika Westerberg wrote:
>>>>> On Wed, Dec 13, 2023 at 08:18:06AM +0200, Mika Westerberg wrote:
>>>>>> On Wed, Dec 13, 2023 at 07:49:14AM +0200, Mika Westerberg wrote:
>>>>>>> On Wed, Dec 13, 2023 at 12:46:35AM +0530, Sanath S wrote:
>>>>>>>> Boot firmware might have created tunnels of its own. Since we cannot
>>>>>>>> be sure they are usable for us. Tear them down and reset the ports
>>>>>>>> to handle it as a new hotplug for USB3 routers.
>>>>>>>>
>>>>>>>> Suggested-by: Mario Limonciello <mario.limonciello@amd.com>
>>>>>>>> Signed-off-by: Sanath S <Sanath.S@amd.com>
>>>>>>>> ---
>>>>>>>>     drivers/thunderbolt/tb.c | 11 +++++++++++
>>>>>>>>     1 file changed, 11 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/drivers/thunderbolt/tb.c b/drivers/thunderbolt/tb.c
>>>>>>>> index fd49f86e0353..febd0b6972e3 100644
>>>>>>>> --- a/drivers/thunderbolt/tb.c
>>>>>>>> +++ b/drivers/thunderbolt/tb.c
>>>>>>>> @@ -2598,6 +2598,17 @@ static int tb_start(struct tb *tb)
>>>>>>>>     	tb_switch_tmu_enable(tb->root_switch);
>>>>>>>>     	/* Full scan to discover devices added before the driver was loaded. */
>>>>>>>>     	tb_scan_switch(tb->root_switch);
>>>>>>>> +	/*
>>>>>>>> +	 * Boot firmware might have created tunnels of its own. Since we cannot
>>>>>>>> +	 * be sure they are usable for us, Tear them down and reset the ports
>>>>>>>> +	 * to handle it as new hotplug for USB4 routers.
>>>>>>>> +	 */
>>>>>>>> +	if (tb_switch_is_usb4(tb->root_switch)) {
>>>>>>>> +		tb_switch_discover_tunnels(tb->root_switch,
>>>>>>>> +					   &tcm->tunnel_list, false);
>>>>>>> Why this is needed?
>>>>>>>
>>>>>>> It should be enough, to do simply something like this:
>>>>>>>
>>>>>>> 	if (tb_switch_is_usb4(tb->root_switch))
>>>>>>> 		tb_switch_reset(tb->root_switch);
>>>> If we don't tear down of tunnels before performing the DPR, the PCIe
>>>> enumeration is failing.
>>>>
>>>> PCIe link is not coming up after DPR. Below log is missing without
>>>> performing path
>>>> deactivation before performing DPR and hence PCIe enumeration is not
>>>> initiated.
>>>>
>>>> [  746.630865] pcieport 0000:00:03.1: pciehp: Slot(0-1): Card present
>>>> [  746.630885] pcieport 0000:00:03.1: pciehp: Slot(0-1): Link Up
>>>>
>>>> I think when we do a DPR, it internally does some handling with PCI Path
>>>> Enable bit(PE).
>>>> So, deactivation of PCIe path is necessary for DPR to work.
>>> Rigth, it should be enough to reset the protocol adapter config and path
>>> config spaces. I guess using discovery at this point is fine too but I
>>> would at least check how complex doing the minimal "reset" turns out.
>>>
>>> I mean in tb_switch_reset() for USB4 v1 routers it can go over all the
>>> adapters and perform "cleanup" or so.
>> I gave it a thought yesterday and we can do something like this:
>>
>> We are already doing tb_discovery(tb) in tb_start. This would
>> discover the path configuration done by Boot firmware.
>>
>> Now, we can place the tb_switch_reset() right below that api with
>> conditions suggested by you.
>>
>> And tb_switch_reset() would internally DPR for all down steam ports.
>>
>> It can look something like below:
>>
>>      /* Find out tunnels created by the boot firmware */
>>          tb_discover_tunnels(tb);
>>      /*
>>       * Reset USB4 v1 host router to get rid of possible tunnels the
>>       * boot firmware created. This makes sure all the tunnels are
>>       * created by us and thus have known configuration.
>>       *
>>       * For USB4 v2 and beyond we do this in nhi_reset() using the
>>       * host router reset interface.
>>       */
>>      if (host_reset && usb4_switch_version(tb->root_switch) == 1)
>>          tb_switch_reset(tb->root_switch);
>>
>> With this, we are making sure while we get a unplug event after doing a DPR,
>> We are clearing all the paths established by Boot firmware. This wouldn't be
>> possible
>> if we had not discovered the paths before we perform DPR.
>>
>> It would create inconsistency for a new hot plug if we have not cleared the
>> path configurations
>> of previous hot unplug events.
> Right. I would still check if doing protocol adapter "reset" + path
> config space clear in tb_switch_reset() is enough and how complex that
> ends up to be. I think that's all what is needed.
>
> If it turns out too complex, yes I guess something like this:
>
> 	/* Find out tunnels created by the boot firmware */
> 	tb_discover_tunnels(tb);
> 	/* Add DP resources from the DP tunnels created by the boot firmware */
> 	tb_discover_dp_resources(tb);
>
> 	if (host_reset && usb4_switch_version(tb->root_switch) == 1) {
> 		struct tb_tunnel *n, *tunnel;
>
> 		list_for_each_entry_safe(tunnel, n, &tcm->tunnel_list, list)
> 			tb_deactivate_and_free_tunnel(tunnel);
>
> 		tb_switch_reset(tb->root_switch);
> 	}
>
> With proper comments would work, no?
Yes, this works. Tested it too and works fine.
Probably we can move tb_deactivate_and_free_tunnel() inside 
tb_switch_reset() to make it
look better.
> Regarding "host_reset", I think we can actually keep it in nhi.c and add
> a parameter to cm_ops->start(reset) that gets passed to the "CM
> implementation". Or something along those lines.
I will check along these lines.
>>>>>> Actually this needs to be done only for USB4 v1 routers since we already
>>>>>> reset USB4 v2 hosts so something like:
>>>>>>
>>>>>> 	/*
>>>>>> 	 * Reset USB4 v1 host router to get rid of possible tunnels the
>>>>>> 	 * boot firmware created. This makes sure all the tunnels are
>>>>>> 	 * created by us and thus have known configuration.
>>>>>> 	 *
>>>>>> 	 * For USB4 v2 and beyond we do this in nhi_reset() using the
>>>>>> 	 * host router reset interface.
>>>>>> 	 */
>>>>>> 	if (usb4_switch_version(tb->root_switch) == 1)
>>>>>> 		tb_switch_reset(tb->root_switch);
>>>>>>
>>>>>> (possibly add similar comment to the nhi_reset() to refer this one).
>>>>> Oh, and would it be possible to tie this with the "host_reset" parameter
>>>>> too somehow? I guess it could be moved to "tb.c" and "tb.h" and then
>>>>> check it from nhi.c as already done and then here so this would become:
>>>>>
>>>>>     	if (host_reset && usb4_switch_version(tb->root_switch) == 1)
>>>>>     		tb_switch_reset(tb->root_switch);
>>>> Is host_reset necessary for USB4 v1 routers ? I did not use host_reset in
>>>> this case.
>>>> If its needed, then we have to modify to enable host_reset in nhi.c as well.
>>> Well you are effectively doing that here, no? You "reset" the host
>>> router therefore tying this to the same command line parameter makes
>>> sense and allows user an "escape hatch" if this turns out breaking
>>> things.