Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7;
Message-ID: <928df647-5b20-406b-8da5-3199f5cfbb48@amd.com>
Date:   Wed, 1 Nov 2023 20:14:31 -0500
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 2/2] PCI: Ignore PCIe ports used for tunneling in
 pcie_bandwidth_available()
To:     Bjorn Helgaas <helgaas@kernel.org>
Cc:     bhelgaas@google.com, mika.westerberg@linux.intel.com,
        andreas.noever@gmail.com, michael.jamet@intel.com,
        YehezkelShB@gmail.com, linux-pci@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-usb@vger.kernel.org,
        Alexander.Deucher@amd.com
References: <20231101225259.GA101390@bhelgaas>
Content-Language: en-US
From:   Mario Limonciello <mario.limonciello@amd.com>
In-Reply-To: <20231101225259.GA101390@bhelgaas>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bUoyczE3OGlCU1g5eDRFd2d4YnhOUGo5UEw5L0hyb1FhQW1ibDN0T2hCQTJw?=
 =?utf-8?B?bnBWbWIvN0prMDRZbUlaRUVhT0JwSUwyekJJeGdOdllVV2x0SmU1WGc1bUI4?=
 =?utf-8?B?RE9NV0R5eFo1K1lUc3ZkTEQrNnpSdlU1cXlXTVRNUlBuaTd0S0RVWnlpWDdW?=
 =?utf-8?B?cFVwbEhqWmQrSE9sUXZ3OE5oN2paWUFzbkF4ZHJVMktjYTB2NG9nTUxSTHhp?=
 =?utf-8?B?Z1czOUxkcUhuZ3hrQ29YV0lvby9nR1haK1Z1YXd6VTZjY3J0SUYvZ1haeVhq?=
 =?utf-8?B?TUhSREYxc1d2Q1RBWStxZFNWSW9qV2ZuV2tTaXNvUjFJejYwa1lHTkZzRUYz?=
 =?utf-8?B?T1dtYnVOWTJJRnhZSjBRbmdRQzJ1ZGtDSTQyMXFoaEhpZTNnQW5lSzR1aWJY?=
 =?utf-8?B?K0pQNXZWR0xJbUFubzNaS2M5VWZqeGVwMGlSRzlCMmRmajcvdW1xZmQ3Q25I?=
 =?utf-8?B?cWR5OVd1ZGlGZjhjbU9veWx3dlBDQzEvWUQ1OTNJd2poWWhacUZHTnQyME15?=
 =?utf-8?B?N1RnWUNsbllwUTRsd0ZLdlNmL3puNW1FMVlSdTRaSHREU1VYMEtZaTFXSERi?=
 =?utf-8?B?ekJWTWJmd0kxQ2tyVWFCKzR0bnFOK1orN0crNjRyYkVLZDdybGtVcGlVcmc0?=
 =?utf-8?B?NHJ2YzVQUnExanFwRnhpaWpENkZwS0hjc211dk14aXBZUEhhNG1YQzkxVVRv?=
 =?utf-8?B?UldvSURvU2E0TlJHWEM2UE1XQlFzN3RsT2huaFFxZVZxa0liNTdJaThXYU4r?=
 =?utf-8?B?YW5WTXZHU1J5V2dqYjFOR2xWT1QzNE9tN2ZncXIzK3daUWJYamVIR05xUzFj?=
 =?utf-8?B?TjJudVF2QklpenByS1BKNEJLeWF1enZlM3Y1SG85ckFNWG5LVjFnS1dmbTNU?=
 =?utf-8?B?VnEwUmJ6NUw1R3RMWFBoK0gyYk5PME1MT2lOU1E5azdRKzhjN2UzR2FTQXZR?=
 =?utf-8?B?T3RFa2FnUzB1OTBacHJsSDluc1dVMVg1ZTcyZ1pnUGJscnpUNVUzbGhwTTRh?=
 =?utf-8?B?RG9Fci9FQXpwTUdLRHZLREZxdEJlWnkrU3JxMWUzQVIrNWYzUlg0MUdMOXBB?=
 =?utf-8?B?MnoraGVZb0lEM3dIR3ZvaThteWxaUldaaTlXWDF1T01yNFY2QTVxdFdGUGYw?=
 =?utf-8?B?SWVHTGZ3RVllK2tpK0xxQ0ROUHFQZjRHY04veEE0NWFQbzJMSG1McGRLNTVp?=
 =?utf-8?B?eXo2UWpHQTlndTR6V3RoYWY4LzFFSjhxZ3lBZHVVOUZ1M0wrOEhnR1F2eSs2?=
 =?utf-8?B?eXk0UERQMVB4TmxYQWg5Tk80Z0hNdGRJMXpkQzVIZzJEM2UrVzk3ajBVQ1RM?=
 =?utf-8?B?Y3dwZHZnenlpaHlVUHY1aWgzUnVRaGpycDE3NzFlT1Z3OWVoajN4a1BMT3NH?=
 =?utf-8?B?Q0N3eUQ2ZkJPUHVnYU5DRmZkTWZKMnJhandXVVg1YzVxb3BzMWhpNUdlZWxM?=
 =?utf-8?B?K0JpVmsxaitBck9uV0k1UEJSNW9XM3lVRHNrZHhTZUdsQS8zSmVmT0U1QndY?=
 =?utf-8?B?bHJ6YVdHSjdqcmloeWRlS0VUcWp5TzdWVEZId0NYMmhMbjFJa1lmaTVIVTdx?=
 =?utf-8?B?MFpUU1NZTVMvK3hOT2QyOFFYdnArZTFiK096S0ttMENpT2hhR2NTS040TEVs?=
 =?utf-8?B?R0lrTnpFYXRXaW9DNXhUSHR5NHhEUXYrTXhOZ0VzNzNYa0FlQlpjaWVpdE5u?=
 =?utf-8?B?elQ2UkhWMFV3SkY0aWVtSHRjaXRobHExYjZvdkY3OFlmTXhMQmp0QnZkL1Fw?=
 =?utf-8?B?TTl1Yk90NG5MZWpYenBoc2FCSmpLNTVRYUpyWHczMXhEa3FwSTlNMVM4ek1i?=
 =?utf-8?B?Y0FwR3N1VkIxa00ycFdSQTJQZ1E4a1E1TTVGZ1diNVg4MDZXMzhqczhmbDZk?=
 =?utf-8?B?bWRNNTl1NHdnSEx0NUxYUG1MaEc1VVhwOVczVHhkdkFHN2EvS3JXUjg0N2Q4?=
 =?utf-8?B?dXpwc0lXNy9mbFFtcjN2b1lUZU53Uko1WHVBR3FEVHZHTzBnRXplUkdGK0U3?=
 =?utf-8?B?TUsrK2FEUDVtTkxsWkZIWjhwcWpXOWFYbVMvYlNJUGlSV0YyUDNpUnZVVW53?=
 =?utf-8?B?TDB5aWxRZEZWRlVGYlpxRlVPNFg3ZjFzcWJNbEJ1ZFd1cllGeGN5VmJuMjli?=
 =?utf-8?Q?1cofZaFOkyouW5JCBxNtVQrCD?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2aea99fc-62a8-4f42-9579-08dbdb411446
X-MS-Exchange-CrossTenant-AuthSource: MN0PR12MB6101.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Nov 2023 01:14:36.5973
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: O2WdCTUIQ4GowMRuLtwNdVA0fMDofchArzKGZqAQg0jBWJK+P94BmRpBiEOJCNdZdTdCWC9qXdxS1/TwjNP71g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR12MB4364
X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 01 Nov 2023 18:14:55 -0700 (PDT)

On 11/1/2023 17:52, Bjorn Helgaas wrote:
> On Tue, Oct 31, 2023 at 08:34:38AM -0500, Mario Limonciello wrote:
>> The USB4 spec specifies that PCIe ports that are used for tunneling
>> PCIe traffic over USB4 fabric will be hardcoded to advertise 2.5GT/s.
>>
>> In reality these ports speed is controlled by the fabric implementation.
> 
> So I guess you're saying the speed advertised by PCI_EXP_LNKSTA is not
> the actual speed?  And we don't have a generic way to find the actual
> speed?

Correct.

> 
>> Downstream drivers such as amdgpu which utilize pcie_bandwidth_available()
>> to program the device will always find the PCIe ports used for
>> tunneling as a limiting factor and may make incorrect decisions.
>>
>> To prevent problems in downstream drivers check explicitly for ports
>> being used for PCIe tunneling and skip them when looking for bandwidth
>> limitations.
>>
>> 2 types of devices are detected:
>> 1) PCIe root port used for PCIe tunneling
>> 2) Intel Thunderbolt 3 bridge
>>
>> Downstream drivers could make this change on their own but then they
>> wouldn't be able to detect other potential speed bottlenecks.
> 
> Is the implication that a tunneling port can *never* be a speed
> bottleneck?  That seems to be how this patch would work in practice.

I think that's a stretch we should avoid concluding.

IIUC the fabric can be hosting other traffic and it's entirely possible 
the traffic over the tunneling port runs more slowly at times.

Perhaps that's why the the USB4 spec decided to advertise it this way? 
I don't know.

> 
>> Link: https://lore.kernel.org/linux-pci/7ad4b2ce-4ee4-429d-b5db-3dfc360f4c3e@amd.com/
>> Link: https://www.usb.org/document-library/usb4r-specification-v20
>>        USB4 V2 with Errata and ECN through June 2023 - CLEAN p710
> 
> I guess this is sec 11.2.1 ("PCIe Physical Layer Logical Sub-block")
> on PDF p710 (labeled "666" on the printed page).  How annoying that
> the PDF page numbers don't match the printed ones; do the section
> numbers at least stay stable in new spec revisions?

I'd hope so.  I'll change it to section numbers in the next revision.

> 
>> Link: https://gitlab.freedesktop.org/drm/amd/-/issues/2925
> 
> This issue says the external GPU doesn't work at all.  Does this patch
> fix that?  This patch looks like it might improve GPU performance, but
> wouldn't fix something that didn't work at all.

The issue actually identified 4 distinct different problems.  The 3 
problems will be fixed in amdgpu which are functional.

This performance one was from later in the ticket after some back and 
forth identifying proper solutions for the first 3.

> 
>> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
>> ---
>>   drivers/pci/pci.c | 41 +++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 41 insertions(+)
>>
>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>> index 59c01d68c6d5..4a7dc9c2b8f4 100644
>> --- a/drivers/pci/pci.c
>> +++ b/drivers/pci/pci.c
>> @@ -6223,6 +6223,40 @@ int pcie_set_mps(struct pci_dev *dev, int mps)
>>   }
>>   EXPORT_SYMBOL(pcie_set_mps);
>>   
>> +/**
>> + * pcie_is_tunneling_port - Check if a PCI device is used for TBT3/USB4 tunneling
>> + * @dev: PCI device to check
>> + *
>> + * Returns true if the device is used for PCIe tunneling, false otherwise.
>> + */
>> +static bool
>> +pcie_is_tunneling_port(struct pci_dev *pdev)
> 
> Use usual function signature styling (all on one line).

OK.

> 
>> +{
>> +	struct device_link *link;
>> +	struct pci_dev *supplier;
>> +
>> +	/* Intel TBT3 bridge */
>> +	if (pdev->is_thunderbolt)
>> +		return true;
>> +
>> +	if (!pci_is_pcie(pdev))
>> +		return false;
>> +
>> +	if (pci_pcie_type(pdev) != PCI_EXP_TYPE_ROOT_PORT)
>> +		return false;
>> +
>> +	/* PCIe root port used for tunneling linked to USB4 router */
>> +	list_for_each_entry(link, &pdev->dev.links.suppliers, c_node) {
>> +		supplier = to_pci_dev(link->supplier);
>> +		if (!supplier)
>> +			continue;
>> +		if (supplier->class == PCI_CLASS_SERIAL_USB_USB4)
>> +			return true;
> 
> Since this is in drivers/pci, and this USB4/Thunderbolt routing is not
> covered by the PCIe specs, this is basically black magic.  Is there a
> reference to the USB4 spec we could include to help make it less
> magical?

The "magic" part is that there is an ACPI construct to indicate a PCIe 
port is linked to a USB4 router.

Here is a link to the page that is explained:
https://learn.microsoft.com/en-us/windows-hardware/design/component-guidelines/usb4-acpi-requirements#port-mapping-_dsd-for-usb-3x-and-pcie

In the Linux side this link is created in the 'thunderbolt' driver.

Thinking about this again, this does actually mean we could have a 
different result based on whether pcie_bandwidth_available() is called 
before or after the 'thunderbolt' driver has loaded.

For example if a GPU driver that called pcie_bandwidth_available() was 
in the initramfs but 'thunderbolt' was in the rootfs we might end up 
with the wrong result again.

Considering this I think it's a good idea to move that creation of the 
device link into drivers/pci/pci-acpi.c and store a bit in struct 
pci_device to indicate it's a tunneled port.

Then 'thunderbolt' can look for this directly instead of walking all the 
FW nodes.

pcie_bandwidth_available() can just look at the tunneled port bit 
instead of the existence of the device link.

> 
> Lukas' brief intro in
> https://lore.kernel.org/all/20230925141930.GA21033@wunner.de/ really
> helped me connect a few dots, because things like
> Documentation/admin-guide/thunderbolt.rst assume we already know those
> details.

Thanks for sharing that.  If I move the detection mechanism as I 
suggested above I'll reference some of that as well in the commit 
message to explain what exactly a tunneled port is.

> 
>> +	}
>> +
>> +	return false;
>> +}
>> +
>>   /**
>>    * pcie_bandwidth_available - determine minimum link settings of a PCIe
>>    *			      device and its bandwidth limitation
>> @@ -6236,6 +6270,8 @@ EXPORT_SYMBOL(pcie_set_mps);
>>    * limiting_dev, speed, and width pointers are supplied) information about
>>    * that point.  The bandwidth returned is in Mb/s, i.e., megabits/second of
>>    * raw bandwidth.
>> + *
>> + * This function excludes root ports and bridges used for USB4 and TBT3 tunneling.
>>    */
>>   u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,
>>   			     enum pci_bus_speed *speed,
>> @@ -6254,6 +6290,10 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,
>>   	bw = 0;
>>   
>>   	while (dev) {
>> +		/* skip root ports and bridges used for tunneling */
>> +		if (pcie_is_tunneling_port(dev))
>> +			goto skip;
>> +
>>   		pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta);
>>   
>>   		next_speed = pcie_link_speed[lnksta & PCI_EXP_LNKSTA_CLS];
>> @@ -6274,6 +6314,7 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,
>>   				*width = next_width;
>>   		}
>>   
>> +skip:
>>   		dev = pci_upstream_bridge(dev);
>>   	}
>>   
>> -- 
>> 2.34.1
>>