Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4;
Message-ID: <f673ffc8-f6f8-4898-d809-effb2c24e53e@amd.com>
Date:   Thu, 14 Dec 2023 12:08:34 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.1
Subject: Re: [Patch v2 2/2] thunderbolt: Teardown tunnels and reset downstream
 ports created by boot firmware
Content-Language: en-US
To:     Mika Westerberg <mika.westerberg@linux.intel.com>
Cc:     Sanath S <Sanath.S@amd.com>, mario.limonciello@amd.com,
        andreas.noever@gmail.com, michael.jamet@intel.com,
        YehezkelShB@gmail.com, linux-usb@vger.kernel.org,
        linux-kernel@vger.kernel.org
References: <20231212191635.2022520-1-Sanath.S@amd.com>
 <20231212191635.2022520-3-Sanath.S@amd.com>
 <20231213054914.GI1074920@black.fi.intel.com>
 <20231213061805.GK1074920@black.fi.intel.com>
 <20231213062306.GL1074920@black.fi.intel.com>
 <adcc6446-8c30-a258-e19b-76fca2c50d21@amd.com>
 <20231213115256.GM1074920@black.fi.intel.com>
From:   Sanath S <sanaths2@amd.com>
In-Reply-To: <20231213115256.GM1074920@black.fi.intel.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?L3Q5RDE5MFVMWnBpam1SRlpkUkhUYm1zYXhKUElkUDNHL0p5Ymg0NHB3WW15?=
 =?utf-8?B?ZGRsZGlPY2FWRGtOQlYyMitrTG5tNEFKd0dRMHpmWVp1VjQzejFFaWtkbmM0?=
 =?utf-8?B?ZFZXNDl4T2tNbkZZQ0doYlZ4ZlltckRjby8wWGxqbVVFYkVqa2JkOFpEb2lQ?=
 =?utf-8?B?OHl0QTNKMUo4aGVkTTZLb3daMWU4K2FOUDFwcm5NQ3N6WWVkYjdpcElCTjFv?=
 =?utf-8?B?cGY2cUxFZVFqUXBzWVNMQlJ3bGtQdXU3THp5a00xTkI2QXE2TXJUMUtMNE9Q?=
 =?utf-8?B?S0N3V3REeUpKSS9wWlQ0NjJ3VXBPUTFmbmFIYXBIa0VNWnhSMDB1Y2hRdkh5?=
 =?utf-8?B?MVU3VmN1R3gwTFM1bklabTNPWW1sMytMVlNDRWdGbWNnR0ZDRk1hNmtXRmgv?=
 =?utf-8?B?bkZoWStrNk9hbG81WndLbU9IN2VsQ1dRTTRrMDdlMVgvOXJWWE9tRitIdkNi?=
 =?utf-8?B?dzJHcDdBcmF6NG43TlB6c0c1T0d1c01PUXZrV1BuZ0p2Skc3WWdETmYxNDlY?=
 =?utf-8?B?K2IyaTJCTldiR0dsR3pyYkkwdVloY3lvVGVWSkhWWUV2TXJpcGt5UHV2Q3ZZ?=
 =?utf-8?B?NVFaSFhpcGROb3hKWTI1TDRaVEd1U0RWMXVZTTV1N2Yzc1Z5a21QWDljRWpn?=
 =?utf-8?B?cklyU2REbGJsWktwdmZxdVpncVc4RjdXUWJCRzl4NnJqZUk2T1EvRE5xUDMx?=
 =?utf-8?B?Q3ZwTDlHZ0NvbVk3ZGlBU2RuVHNnTjFOb01oMTcrRVMzWmg0M0V4Vkt0VkJl?=
 =?utf-8?B?d2hhK01JTElWenBqVHhwZTFSMmtxNEhWWUlBb2FuVWdyTXZMeGQ5L1ZydXVt?=
 =?utf-8?B?Q1pxME83eUhzMlUvWFY4U3JyOFdWRmNmeXNSblJaV0ozeUI3ZFNOcXp2ODgw?=
 =?utf-8?B?MEJRVVY4bzREVnJUOFpMejUwZXJRWk5qQUxNMlNJTWd2b3ZLTGZhY3pVNTlv?=
 =?utf-8?B?SXBQcW5aUXhzSTFRUTR5SktqQ3F3U251OThFV3dZZklzZjFTT1dyT3pyKzJP?=
 =?utf-8?B?MWU3Z2E0ZitYMHF6S3pNc2Y5TmdIQmUvWjdVYWdqeURTYzVKUkNlUmh0aVV5?=
 =?utf-8?B?U3BkUXRlbEU0YnBFYWFIU0hBenVHbE9sdWx4K0k3cng1Y3FiZHJNWWtKZkUw?=
 =?utf-8?B?cTIxM1dTUTgvT28zZlNiamZZbWFSV1EwYS9RQ2gzTXJzWGNjL0IyZU44anZ2?=
 =?utf-8?B?T253cjd1VGZXcHplOG5UTmN0M2REOTZBT1dGV1cwQjRlSWxjblJtTFlJb0hC?=
 =?utf-8?B?M2VJQ0Jkc3hxdm1mUklZVWRQaTR2VTluQXNXa040UmdkTkYzYUZCeVd3RW1J?=
 =?utf-8?B?YmxMNU1JZFoxdjFoRFJONWt3Yk55cE5DT25RMDE0MVRHckQxVytoemtWNE1V?=
 =?utf-8?B?dGhMbjN2VW8vQllQSDh0NWJJaFZlL24yZnEvbnBFNk8ySjQ1SHkzN2ZITkhI?=
 =?utf-8?B?a3Q4LzRuaHJJZi9IWUgxeGJrNk1uMDNSeXhraEF6WGQwRDB4bTR3R0k5WWQw?=
 =?utf-8?B?L2FkMmxuazkxcjdkZzZGSjJCZEhwNUFHSVR6Rkg5aUt6L09zNE9iYXVVdzlj?=
 =?utf-8?B?bzJyTFliUGVLOGkwdDBiR3hPc2tMOXRSUHVJMnZPbEk1dkpIVEllOG9sZTZo?=
 =?utf-8?B?SEZYT3ZiTW5xM3h2UTNTNGZNRVZpcU1LNVFXR0t1dng1NW5zWU8xaVdVSnBN?=
 =?utf-8?B?S241TDZKcTUxYnRQa3N3a0t0dVpJdGFPOUxZS0FvSnB6QXlpOEJOT282RkFz?=
 =?utf-8?B?T3pXMVlLRk93cTB3UHB4eXUvVE1hTlU0bThNM0NYeXN1blN3Ky9jMGpGcmxB?=
 =?utf-8?B?VU96RGV3dEtoNExtelVjNWxKcXdPcERkZEYwcittdjRYK0R5dnVQWjUvL2dw?=
 =?utf-8?B?N3hlYnJaSXRubElsc2lZOFFjbjBHTWgxZCs3YXJYNlpiSk1JMWs2SEhlMW1I?=
 =?utf-8?B?NHpYT2tEU0o0Y0YxOW8wSmFQUUhTc0plcWtqRk94eXNDcG1oVUttbHFYMkFz?=
 =?utf-8?B?TURsbjVScW0xeWM4Y0F1NGV3MjhQMzlRYlpUS0pjMmxGTmxKUWM5K29zSWhW?=
 =?utf-8?B?MStYRHVyWkNsdzVOdkd3OUQ5akJYRzlVbUVHVjI5Vm5pNk1DeVpvaHdXZTB5?=
 =?utf-8?Q?nTezvCFTHHG2D0y83+KrEWNND?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7f89361c-1339-4318-26a0-08dbfc6f509b
X-MS-Exchange-CrossTenant-AuthSource: MW4PR12MB7016.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Dec 2023 06:38:43.0057
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: da6uMoe9hSUFPjOhl/cdIPQzxwdG2w1DxLtJuNTi3miqadCzoIGOxOq+p6Gd0utdnN13yyH2tFvWTdFRpEJ34w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6354
X-Spam-Status: No, score=-1.6 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 13 Dec 2023 22:39:01 -0800 (PST)


On 12/13/2023 5:22 PM, Mika Westerberg wrote:
> On Wed, Dec 13, 2023 at 04:04:57PM +0530, Sanath S wrote:
>> On 12/13/2023 11:53 AM, Mika Westerberg wrote:
>>> On Wed, Dec 13, 2023 at 08:18:06AM +0200, Mika Westerberg wrote:
>>>> On Wed, Dec 13, 2023 at 07:49:14AM +0200, Mika Westerberg wrote:
>>>>> On Wed, Dec 13, 2023 at 12:46:35AM +0530, Sanath S wrote:
>>>>>> Boot firmware might have created tunnels of its own. Since we cannot
>>>>>> be sure they are usable for us. Tear them down and reset the ports
>>>>>> to handle it as a new hotplug for USB3 routers.
>>>>>>
>>>>>> Suggested-by: Mario Limonciello <mario.limonciello@amd.com>
>>>>>> Signed-off-by: Sanath S <Sanath.S@amd.com>
>>>>>> ---
>>>>>>    drivers/thunderbolt/tb.c | 11 +++++++++++
>>>>>>    1 file changed, 11 insertions(+)
>>>>>>
>>>>>> diff --git a/drivers/thunderbolt/tb.c b/drivers/thunderbolt/tb.c
>>>>>> index fd49f86e0353..febd0b6972e3 100644
>>>>>> --- a/drivers/thunderbolt/tb.c
>>>>>> +++ b/drivers/thunderbolt/tb.c
>>>>>> @@ -2598,6 +2598,17 @@ static int tb_start(struct tb *tb)
>>>>>>    	tb_switch_tmu_enable(tb->root_switch);
>>>>>>    	/* Full scan to discover devices added before the driver was loaded. */
>>>>>>    	tb_scan_switch(tb->root_switch);
>>>>>> +	/*
>>>>>> +	 * Boot firmware might have created tunnels of its own. Since we cannot
>>>>>> +	 * be sure they are usable for us, Tear them down and reset the ports
>>>>>> +	 * to handle it as new hotplug for USB4 routers.
>>>>>> +	 */
>>>>>> +	if (tb_switch_is_usb4(tb->root_switch)) {
>>>>>> +		tb_switch_discover_tunnels(tb->root_switch,
>>>>>> +					   &tcm->tunnel_list, false);
>>>>> Why this is needed?
>>>>>
>>>>> It should be enough, to do simply something like this:
>>>>>
>>>>> 	if (tb_switch_is_usb4(tb->root_switch))
>>>>> 		tb_switch_reset(tb->root_switch);
>> If we don't tear down of tunnels before performing the DPR, the PCIe
>> enumeration is failing.
>>
>> PCIe link is not coming up after DPR. Below log is missing without
>> performing path
>> deactivation before performing DPR and hence PCIe enumeration is not
>> initiated.
>>
>> [  746.630865] pcieport 0000:00:03.1: pciehp: Slot(0-1): Card present
>> [  746.630885] pcieport 0000:00:03.1: pciehp: Slot(0-1): Link Up
>>
>> I think when we do a DPR, it internally does some handling with PCI Path
>> Enable bit(PE).
>> So, deactivation of PCIe path is necessary for DPR to work.
> Rigth, it should be enough to reset the protocol adapter config and path
> config spaces. I guess using discovery at this point is fine too but I
> would at least check how complex doing the minimal "reset" turns out.
>
> I mean in tb_switch_reset() for USB4 v1 routers it can go over all the
> adapters and perform "cleanup" or so.
I gave it a thought yesterday and we can do something like this:

We are already doing tb_discovery(tb) in tb_start. This would
discover the path configuration done by Boot firmware.

Now, we can place the tb_switch_reset() right below that api with
conditions suggested by you.

And tb_switch_reset() would internally DPR for all down steam ports.

It can look something like below:

     /* Find out tunnels created by the boot firmware */
         tb_discover_tunnels(tb);
     /*
      * Reset USB4 v1 host router to get rid of possible tunnels the
      * boot firmware created. This makes sure all the tunnels are
      * created by us and thus have known configuration.
      *
      * For USB4 v2 and beyond we do this in nhi_reset() using the
      * host router reset interface.
      */
     if (host_reset && usb4_switch_version(tb->root_switch) == 1)
         tb_switch_reset(tb->root_switch);

With this, we are making sure while we get a unplug event after doing a DPR,
We are clearing all the paths established by Boot firmware. This 
wouldn't be possible
if we had not discovered the paths before we perform DPR.

It would create inconsistency for a new hot plug if we have not cleared 
the path configurations
of previous hot unplug events.
>>>> Actually this needs to be done only for USB4 v1 routers since we already
>>>> reset USB4 v2 hosts so something like:
>>>>
>>>> 	/*
>>>> 	 * Reset USB4 v1 host router to get rid of possible tunnels the
>>>> 	 * boot firmware created. This makes sure all the tunnels are
>>>> 	 * created by us and thus have known configuration.
>>>> 	 *
>>>> 	 * For USB4 v2 and beyond we do this in nhi_reset() using the
>>>> 	 * host router reset interface.
>>>> 	 */
>>>> 	if (usb4_switch_version(tb->root_switch) == 1)
>>>> 		tb_switch_reset(tb->root_switch);
>>>>
>>>> (possibly add similar comment to the nhi_reset() to refer this one).
>>> Oh, and would it be possible to tie this with the "host_reset" parameter
>>> too somehow? I guess it could be moved to "tb.c" and "tb.h" and then
>>> check it from nhi.c as already done and then here so this would become:
>>>
>>>    	if (host_reset && usb4_switch_version(tb->root_switch) == 1)
>>>    		tb_switch_reset(tb->root_switch);
>> Is host_reset necessary for USB4 v1 routers ? I did not use host_reset in
>> this case.
>> If its needed, then we have to modify to enable host_reset in nhi.c as well.
> Well you are effectively doing that here, no? You "reset" the host
> router therefore tying this to the same command line parameter makes
> sense and allows user an "escape hatch" if this turns out breaking
> things.