Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1;
Message-ID: <57b2b450-2fc9-ac68-d016-fb1d061ba992@amd.com>
Date:   Tue, 12 Sep 2023 15:39:33 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.11.0
Subject: Re: [PATCH v4 2/3] PCI: Enable support for 10-bit Tag during device
 enumeration
Content-Language: en-US
To:     Lukas Wunner <lukas@wunner.de>
Cc:     linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org,
        Bjorn Helgaas <bhelgaas@google.com>,
        Mahesh J Salgaonkar <mahesh@linux.ibm.com>,
        Kuppuswamy Sathyanarayanan 
        <sathyanarayanan.kuppuswamy@linux.intel.com>,
        Yazen Ghannam <yazen.ghannam@amd.com>
References: <20230815212043.114913-1-Smita.KoralahalliChannabasappa@amd.com>
 <20230815212043.114913-3-Smita.KoralahalliChannabasappa@amd.com>
 <20230828095429.GA17864@wunner.de>
From:   Smita Koralahalli <Smita.KoralahalliChannabasappa@amd.com>
In-Reply-To: <20230828095429.GA17864@wunner.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bzFLMzVoaVNvU0dqT2hsRnZNclpPeERMbm1INk9XTnF5TGQxc0ZWRUdGbitF?=
 =?utf-8?B?R0tDOGNhOGZVMUVWcGhCM2FRVWo4c1NlbitMZHEweGNQeEFBd0s2UGZFMWlV?=
 =?utf-8?B?c3ZKUTF1Q0l3cndlWFRmV0FzRWtQNUFpMVU1ZTJRY3YvUEp1bEk0V2Q2RGVS?=
 =?utf-8?B?aEQ1SjdNbU85N1lPM3NmM0VVS3JGRzN5dHdTNVVHcGFtMDNwVnhjc0hTeFYw?=
 =?utf-8?B?K3l0THhFcGJNaG5VNExqWW9jUkx3RkZmeDJ1ZXpzbGpkdUZjaHJJZm4zM21q?=
 =?utf-8?B?K3Evcm9sdkVVZHR4dlpSRUx4SjBkU1BCZHRuMG5BQmVSMDBOWkJjdUJsYlYy?=
 =?utf-8?B?UVFzMEczNE5lRFVkazhnZVY0a2xyVkQrOE1OOGdDUkNtUUQvczNsZ1hScFFN?=
 =?utf-8?B?bnJhUzJwTlhRbnM1KzlsRFh5VnRsTWVLYm1JY0xvZHdUSjBvcHBnU3FiOGFR?=
 =?utf-8?B?OGRnVWRJdUdSRjdwMVlyMjdtY00yS0l0a1N0MDVxYjc3RUpGb0hKZ3RsSE9F?=
 =?utf-8?B?d3lxM0g3R2paaDNiM3gyS0tnV3Zyamg4Z0dmcFBWVTRpaStVSFhLRDc3eFBO?=
 =?utf-8?B?dEJhdnBBZlNaOUtUYVRNYlcrMytPcDAweE1TRWl4OHNkems2cHZ1OEpqVEwv?=
 =?utf-8?B?eVJhMlErK3oxbUhQTHgyNEorMzVOeDgxRFVoRzg2YUhPRzFEU1U0VVJaM2dW?=
 =?utf-8?B?U3pDMVFrWmFHaFRNTnB0TFBOOUJRZS84Q3gxSzV2Y0hoOXZRUWJhTkE0MGpB?=
 =?utf-8?B?ZlE4QXM5ZXIzZWd4YnowdmZVQTdCZGRCK0JZT0Q0eTZMNnFOOGlXYWR4TXJ6?=
 =?utf-8?B?Y3N5bTQ1ZUxPaWl4WnlLamltSHpiTXB4UTFpQjlnUHFjcTQ5ZG5OTDd0bGMx?=
 =?utf-8?B?Nlo2ZU1lOHgvTk9iblpNVEVOeC9zQWs2cy8ybEtyZTYycmh0SU9qU1BBa0Q3?=
 =?utf-8?B?aExCMjV0UHNjRHhDczRRWUpOaExINTZQWTVCMU5lV0FCNTZCekx0emh0ZkZi?=
 =?utf-8?B?RStocHpmT1pRK3hFL2xUa0MxczczbWE0c1Q5WG1TVnpkZGgwTGczMDdEbmdp?=
 =?utf-8?B?ZlVxVTNRcUg0RHdlU1pQWlU3SHBaVERybk5iM3lOSG13QXB5U2tzaU5LTk5B?=
 =?utf-8?B?Qy9sK1FVd09KYlR2MkUvbFNsOUZudExnWjExb3h5ZU15VnlGdXlHbUsyN09y?=
 =?utf-8?B?UkptbnNmWHV3NmsyWmNVcnhRZ0ZmYWRGd2dIUDJqTlBJWndCUWJaTnpsNWdi?=
 =?utf-8?B?VmdvOFFIcEtUNkZCSmpkdkcwa05vV3o1dEpEUHBtRUJCNXJQcWRVR2U1aWRQ?=
 =?utf-8?B?cjJCRlpVRmh3MDRTblBBM1l1NFRJZ1dXcUJpa2cxM2hCUWhuUXlpNEEvYVZQ?=
 =?utf-8?B?NnhmVFlQTE1qNW51dGdhVmJ3YS9yUDk4SndZdU5ERitTcGhUbzV1R3FTUFI3?=
 =?utf-8?B?N0hUWm9HcHFib2dHUEpBU2E4OGJRcnBGcW1aWWpoY3AvRW9SZjRXdXdwQko3?=
 =?utf-8?B?c2FWZmZybUVRRUltVXl6TjcrY0dONm9kdlcvZ2o5UHkyL2NiMEUxZmx3RWh3?=
 =?utf-8?B?N285UFhRc3U3TXFteE5jN1F6MDA1MnBXbU9pb25tUjMwK0pvU3dPTEJPdlF2?=
 =?utf-8?B?aVFvM1hNNTFiQ2xOa2t2aDJ4cVI4eGJaOXBQK28zYVE2dzNncWloc1gzUlEx?=
 =?utf-8?B?M2hjNktXdG5xZlMzeUVCSW04aWx6TzlRMDNSWmdNS1VqTWxoc0U5RzlEMFd5?=
 =?utf-8?B?cEN3RDhRUlRXSVp4YW9XKzROcTRhdS9DYk1PSGdHTzVXaEdXaGdIUmVBTkNS?=
 =?utf-8?B?NEI2enZLVkR0aXcxZ2F0alJiRzBiNnpiTi91bEp6ZndLdE82UllOMm1JaU9H?=
 =?utf-8?B?eFNuNU1OTGpSR1R2L1NvMGNISU55QXJJY2xSVEUzb0M3MTVzVXdsdHhybFp4?=
 =?utf-8?B?Yi9Td0wraXlmemhaVThFY0hDM0RzTWNuSW45VkxCdldodXRKYzVZdEpvVi9P?=
 =?utf-8?B?M3FRS3l6eXBTUWhFcTZiYVh0M21ZVjZDNDh4a2MrcFdhSlNDWm96OURBbDJN?=
 =?utf-8?B?aE1NRi9mM1lzRWNLYTdqRGlvT0VBaGdQN2xBYmNMeWRIUDltZ2ppakR3a3dW?=
 =?utf-8?Q?CNo/DZHog4HGbySxrvlErVSni?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 541c3f33-d72d-4c88-a009-08dbb3e1243e
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB2869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Sep 2023 22:39:36.1926
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JV0yvIIA0RWfPZq/FnLiZ+jKsAOPKGBlntCLGrzLeUV+ARUkYQW1NNdxFbl6xkXQwkh530yEA/VuhOFAOVb/pg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR12MB6545
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Tue, 12 Sep 2023 15:40:25 -0700 (PDT)
X-Spam-Status: No, score=-2.3 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email

On 8/28/2023 2:54 AM, Lukas Wunner wrote:
> On Tue, Aug 15, 2023 at 09:20:42PM +0000, Smita Koralahalli wrote:
>> +void pci_configure_ten_bit_tag(struct pci_dev *dev)
>> +{
>> +	struct pci_dev *bridge;
>> +	u32 cap;
>> +
>> +	if (!pci_is_pcie(dev))
>> +		return;
>> +
>> +	bridge = dev->bus->self;
>> +	if (!bridge)
>> +		return;
> 
> I think you need to use bridge = pcie_find_root_port(dev) because
> "dev" may be further down in the hierarchy with several switches
> in-between it and the Root Port.
> 
> Note that pcie_find_root_port(dev) returns NULL if !pci_is_pcie(dev),
> so the check above may become unnecessary.
> 
> If pcie_find_root_port(dev) == dev, then dev itself is a Root Port,
> in which case you need to bail out.

Will fix thanks!

> 
> 
>> +	/*
>> +	 * According to PCIe r6.0 sec 7.5.3.15, Requester Supported can only be
>> +	 * set if 10-Bit Tag Completer Supported bit is set.
>> +	 */
>> +	pcie_capability_read_dword(bridge, PCI_EXP_DEVCAP2, &cap);
>> +	if (!(cap & PCI_EXP_DEVCAP2_10BIT_TAG_COMP))
>> +		goto out;
>> +
>> +	if (cap & PCI_EXP_DEVCAP2_10BIT_TAG_REQ) {
> 
> Hm, if Requester Supported cannot be set unless Completer Supported is
> also set, why check for Completer Supported at all?

Makes sense to me. Will change.
> 
> 
>> --- a/drivers/pci/probe.c
>> +++ b/drivers/pci/probe.c
>> @@ -2476,6 +2476,7 @@ static void pci_init_capabilities(struct pci_dev *dev)
>>   	pci_pm_init(dev);		/* Power Management */
>>   	pci_vpd_init(dev);		/* Vital Product Data */
>>   	pci_configure_ari(dev);		/* Alternative Routing-ID Forwarding */
>> +	pci_configure_ten_bit_tag(dev); /* 10-bit Tag Requester */
>>   	pci_iov_init(dev);		/* Single Root I/O Virtualization */
>>   	pci_ats_init(dev);		/* Address Translation Services */
>>   	pci_pri_init(dev);		/* Page Request Interface */
> 
> Hm, isn't this too late to disable 10-bit tags if a hot-plugged device
> doesn't support it?  There are plenty of config space reads/writes
> happening before pci_configure_ten_bit_tag() and if the Root Port
> has 10-bit tags enabled by BIOS because a previously unplugged
> device supported it, I assume the Root Port may use 10-bit tags for
> those config space accesses, despite the newly hotplugged device not
> supporting them?
> 
> If so, you may indeed have to unconditionally disable 10-bit tags
> upon device removal and re-enable them once a 10-bit capable device
> is hotplugged.
> 
> I'm wondering what happens if there are switches between the hotplugged
> device and the Root Port.  In that case, there may be further devices
> in the hierarchy below the Root Port.  I assume 10-bit tags can only be
> enabled if *all* devices below the Root Port support them, is that correct?

You are right! I understand I missed the consideration of involving 
switches and hierarchical PCIe structures.

> 
> The corollary would be that if there's an unoccupied hotplug port somewhere
> in the hierarchy below a Root Port, 10-bit tags cannot be enabled at all
> on the Root Port.

Yes, but the BIOS might have already enabled 10-bit tags on root port 
before this hotplug port becomes unoccupied on hot-remove.

   Maybe we can leave 10-bit tags enabled on hot-removal
> and only disable them on hot-add?

Considering all the challenges, would you think we should 
unconditionally clear 10-bit tags on remove and enable them on add? 
Because the TLPs issue will exist even if we leave the tags enabled on 
removal. Disabling unconditionally would atleast resolve config space 
read/writes and p2pdma issues. What do you think?

  That wouldn't work however if TLPs
> are sent to the hot-added device without operating system involvement
> prior to enumeration by the operating system.  Don't CXL devices
> autonomously send PM messages upstream on hot-add?
> 
> There's another quagmire:  Endpoint devices may talk to each other via
> p2pdma (see drivers/pci/p2pdma.c) and if either of them doesn't support
> 10-bit tags, we need to disable 10-bit tags on them upon commencing
> p2pdma.  We may re-enable 10-bit tags once either of the devices is
> hot-removed or p2pdma between them is stopped.
> 
> Finally, PCIe r6.0 added 14-bit tag support.  It may be worth adding
> 10-bit tag support in a way that 14-bit tag support can easily be added
> later on (or is added together with 10-bit tag support).

I agree.

Thanks,
Smita

> 
> Thanks,
> 
> Lukas