Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <9d1095ab-23e5-3df3-58d6-b2974f87ee72@amd.com>
Date:   Wed, 5 Jul 2023 23:07:34 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.12.0
Subject: Re: [Intel-wired-lan] [PATCH] PCI/ASPM: Enable ASPM on external PCIe
 devices
Content-Language: en-US
To:     Bjorn Helgaas <helgaas@kernel.org>,
        Kai-Heng Feng <kai.heng.feng@canonical.com>
Cc:     Kuppuswamy Sathyanarayanan 
        <sathyanarayanan.kuppuswamy@linux.intel.com>,
        linux-pci@vger.kernel.org,
        "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>,
        linux-kernel@vger.kernel.org, Vidya Sagar <vidyas@nvidia.com>,
        Michael Bottini <michael.a.bottini@linux.intel.com>,
        intel-wired-lan@osuosl.org, bhelgaas@google.com,
        Mika Westerberg <mika.westerberg@linux.intel.com>
References: <20230705200617.GA72825@bhelgaas>
From:   Mario Limonciello <mario.limonciello@amd.com>
In-Reply-To: <20230705200617.GA72825@bhelgaas>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cm4xUnVhcExkN0xGZkl0ZTQzSCtzSDR5b1R4dFQ0S1JWZEJlRUtIVzMzeDYw?=
 =?utf-8?B?ZnVUZXU3d2FXMENNRWFIWnlmalJyTm1kdUlaNStGNU9iMnBsZWRZQnVveHJL?=
 =?utf-8?B?L1IyakI3UFJ6UHhEU0sxdUpFK3EwSDk3ZUVlZFpYc1UzTFo4L2t4b2p5VjB1?=
 =?utf-8?B?WEpIMWdia3RZSmlpcDlPVWZ0WXVnZ1RvTlAxYzdSOFBWb0J3M0pyREFDUXA1?=
 =?utf-8?B?WEwxMGJHTDBDcC90bk1YZTRkQlpua0I4dHM3dkZmWGZneDIrZms1T1FWVDk2?=
 =?utf-8?B?Qk5veU1iczhtTCtRRHY2NndaU3dvVXNnWlA0WTRBTFNSb0FDNy8zV3d0aitE?=
 =?utf-8?B?WkZjeUNtK2hTTlJ1SjErai81Q1JQaGJRQ2tLYnZZSDYwN29lZGJnZWhIQURw?=
 =?utf-8?B?UU0xcGE5dDk1WGJqM1NSdmU1WGNLa1Rlemk2bGExeWpJTm8zVTJTbkFRVEZi?=
 =?utf-8?B?dzRmMWdSYkE0a2kxYmNwaENBb2p0eUF3dURybzdyTm1PTjloZEVlK01nRnZp?=
 =?utf-8?B?ZDRIYzRkOHcvYTk1dThGSWZiRGJTaXE4MTcySGdXNE9pMnV6OVphVy9kZHhM?=
 =?utf-8?B?WFVqTWpUbGpDTWV6T0xrWXEza0ZUUWxCNCtvZldsbTFWcitKZ2ZwV1BUVkM1?=
 =?utf-8?B?WjFvTmNwWUUrUjFmbG9sT21VbkluaEVMWDEzTDVUNUNIQjdtcUJkY1YxZy8z?=
 =?utf-8?B?VDd3N0lpVGRza2VxZE1zakZITi9ZRzBleUhiOGpyZjR0M1ZkVzlCOXpsb1RP?=
 =?utf-8?B?czVzYTdFM2R3d09YbW4xcTlOVjAremJNWVYzZTNMcW1jM2RvL25kQkg3SnZw?=
 =?utf-8?B?SnlPRjB1V29mK3N1Rnkwb3MxNmZFSmc4QXBubEkxVGRwY0h2WGM5OU1uNC9n?=
 =?utf-8?B?TnkyQ1lpR0pPR2FtNjhUdTZUaDNDTFlDM3ZVOWtCY01ycjRmb0ZVUUNINExp?=
 =?utf-8?B?ODAxZFhtNjVvZlQzdjNEUXBaaGVYclpqOVhvVlJoVWgvVkEremNld2U5elZK?=
 =?utf-8?B?bEpHV25pbkx1NTF4UDJJcVdYNW4xcTBHZTdkeUY2cnRTemhUamU4cy94dU9k?=
 =?utf-8?B?TnNBdDltYXd2cTB3clE1dHVNbHlZbWhIdXc2Y2hNb0NFaW11L0JyK3BlMGFG?=
 =?utf-8?B?L0h2NlZjaklYV2QrUUNWVW9HV01BbXc0KzloUWZFbzh1a2VNNHdjSS8zWVRy?=
 =?utf-8?B?ZVFyYWNJNVBLTU51QTloN09Oa2d3VWkxcjJ4TGRtNHMvRnJ3OVJ6V0RwdnpO?=
 =?utf-8?B?cHRpUllkcGFZZDVNWFZkaWpuUEZJK2FubVh2MzRmYWxlUUlqQnFEQ1p2blpj?=
 =?utf-8?B?K3R1bHEwWHduakR2ZFI3NUQ5VmtSZFV0NWlDa2RWK0VkRTczUk1kY1RQREpt?=
 =?utf-8?B?VWhzVHo0MDdTVVh2Z3lsSG9CWE5sSk5FMFFadDBKblZYQTFYaWNiT1lKK2hu?=
 =?utf-8?B?UCtwb2NpcHdqdnR4WjIxS3ZmREZPbS9wRTF2c0t0dkFwQmlMTk9SZzlocEww?=
 =?utf-8?B?TXdMUFVTTlFRanFzZEs0RjNhT0IwOEZFQm1WZC8xMHNDT3ZqVUJ1Mnlnb3dh?=
 =?utf-8?B?QThTR1VQang0NWlBODc3QXltS0FDMFhjUkFHZG1zcDhKZjJTUGd0VnIvVFVN?=
 =?utf-8?B?Vy9kdE92ZENuN1Y5cGx3OFAxSGNWbURIMm1xNVhQelZKZVNNdEkvc21zQnZw?=
 =?utf-8?B?TklGRmVOelRTSEZNRTgzbmowTmtmam5yUncrdHFMT0FJVWVjWmZLTThIOXVq?=
 =?utf-8?B?ajRuNzMzQTloMUhqQ1VCUnQwYVZaR2NHbUJTeC96cDlMK1M5SXJjQ1lVekhx?=
 =?utf-8?B?b293Z2VyVjRmWnA4eENva09TUmwxY0VoWGFnZzBtYUVVRk5rdTc0NUUxK21x?=
 =?utf-8?B?L0w5Ym1wY28yQlptMFVaUER6cXUwL2F6TzlGTU5zZnUvR25NYWE2VzdyZjJU?=
 =?utf-8?B?ZVVyd3pjNERvNDg4eVpoZkxZUWxBVnI4WHFSS1A5d2g5S2Z4aWJLbXlZVDdr?=
 =?utf-8?B?NXo0RERVcWpLelJRMDdBVFZpd3JwZmxhZnl3T3RITS9GbXFleVBkWmh0ekFO?=
 =?utf-8?B?enRKMWJXeVJIMkIrWWpuellxMXFjMFhSMy9wNVRHR25XcUlnbVlHcWkvWWNu?=
 =?utf-8?B?YWUzdW9nU2s2ZlhGclRja2NEU3o5UktSTWRrdmZ3QjBhdVlPeUpNS0dQWGZL?=
 =?utf-8?B?TVE9PQ==?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 04d3ec36-339e-4267-accc-08db7dd6883d
X-MS-Exchange-CrossTenant-AuthSource: MN0PR12MB6101.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jul 2023 04:07:36.6923
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: fWXceW1yqZs3t43jzJtUG6uA0LuCS+zztMP+q6Ex97oALaERmC8mz2w21N2G1Jss4PIf95bYK8qKNzwKpbicbw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6045
X-Spam-Status: No, score=-1.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        SPF_HELO_PASS,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=no
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 7/5/23 15:06, Bjorn Helgaas wrote:
> On Wed, Jun 28, 2023 at 01:09:49PM +0800, Kai-Heng Feng wrote:
>> On Wed, Jun 28, 2023 at 4:54 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>>> On Tue, Jun 27, 2023 at 04:35:25PM +0800, Kai-Heng Feng wrote:
>>>> On Fri, Jun 23, 2023 at 7:06 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>>>>> On Tue, Jun 20, 2023 at 01:36:59PM -0500, Limonciello, Mario wrote:
> 
>>> It's perfectly fine for the IP to support PCI features that are not
>>> and can not be enabled in a system design.  But I expect that
>>> strapping or firmware would disable those features so they are not
>>> advertised in config space.
>>>
>>> If BIOS leaves features disabled because they cannot work, but at the
>>> same time leaves them advertised in config space, I'd say that's a
>>> BIOS defect.  In that case, we should have a DMI quirk or something to
>>> work around the defect.
>>
>> That means most if not all BIOS are defected.
>> BIOS vendors and ODM never bothered (and probably will not) to change
>> the capabilities advertised by config space because "it already works
>> under Windows".
> 
> This is what seems strange to me.  Are you saying that Windows never
> enables these power-saving features?  Or that Windows includes quirks
> for all these broken BIOSes?  Neither idea seems very convincing.
> 

I see your point.  I was looking through Microsoft documentation for 
hints and came across this:

https://learn.microsoft.com/en-us/windows-hardware/customize/power-settings/pci-express-settings-link-state-power-management

They have a policy knob to globally set L0 or L1 for PCIe links.

They don't explicitly say it, but surely it's based on what the devices 
advertise in the capabilities registers.

>>>> So the logic is to ignore the capability and trust the default set
>>>> by BIOS.
>>>
>>> I think limiting ASPM support to whatever BIOS configured at boot-time
>>> is problematic.  I don't think we can assume that all platforms have
>>> firmware that configures ASPM as aggressively as possible, and
>>> obviously firmware won't configure hot-added devices at all (in
>>> general; I know ACPI _HPX can do some of that).
>>
>> Totally agree. I was not suggesting to limiting the setting at all.
>> A boot-time parameter to flip ASPM setting is very useful. If none has
>> been set, default to BIOS setting.
> 
> A boot-time parameter for debugging and workarounds is fine.  IMO,
> needing a boot-time parameter in the course of normal operation is
> not OK.
> 
> Bjorn