Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <b1933ca4-13a8-83a0-d9f3-1ecea467948d@nvidia.com>
Date:   Tue, 13 Sep 2022 18:45:43 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.3.2
Subject: Re: [PATCH V3] PCI/ASPM: Save/restore L1SS Capability for
 suspend/resume
Content-Language: en-US
To:     Bjorn Helgaas <helgaas@kernel.org>
Cc:     bhelgaas@google.com, lorenzo.pieralisi@arm.com,
        refactormyself@gmail.com, kw@linux.com, rajatja@google.com,
        kenny@panix.com, kai.heng.feng@canonical.com, treding@nvidia.com,
        jonathanh@nvidia.com, abhsahu@nvidia.com, sagupta@nvidia.com,
        benchuanggli@gmail.com, linux-pci@vger.kernel.org,
        linux-kernel@vger.kernel.org, kthota@nvidia.com,
        mmaddireddy@nvidia.com, sagar.tv@gmail.com
References: <20220907210540.GA140988@bhelgaas>
From:   Vidya Sagar <vidyas@nvidia.com>
In-Reply-To: <20220907210540.GA140988@bhelgaas>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bTMyQVZDZmQzcExIQzN4dWVaM0luMmZpVU5URUx2NTUvMnhYOUlVcFNhQTky?=
 =?utf-8?B?RlpHeVJxT0lpaUVpdU5VM3NncVN6L21VMUpoN0Rlb3hKMUNNL0pNTm9rNnJT?=
 =?utf-8?B?SkwzZ3o2TFlvbkFOc0M3bXNiOUhaN0wyM0krbTl3cm1Bc0F6WGp2Qnl4M3FK?=
 =?utf-8?B?SU5WOGxxNktod3huTVh0b2FUY1M0MHhhUjZGcmt6UHdEeGtGVDNJYXJSN1kz?=
 =?utf-8?B?N3BjTjNOVTF1L3BhY2NRZWQ2elNzU01mQVlHemtuaHY4VmEwQ2tWOE5DVUR2?=
 =?utf-8?B?SUp5RHYyd2ZRVU5Fem1OZU03TEVTNlo0MElXZm5UT0doVzgzMEZ4WTJPQ3R4?=
 =?utf-8?B?Tmgwd2lDVFhIM2RHNzZ5a0dlNkg1b3dMcDlFRXpqN2NQM2FGRnYwRXlXUlc0?=
 =?utf-8?B?UHl3aDlVTjBIckdxNUxrYU5tZWZMeENQZWt1a0RObXBaSDV2a2tpbW5QNDZD?=
 =?utf-8?B?cDV0U3k0Sjl5SlNGdGticFdENlVUNVNMVjAyRGowYi9jT2ViTDExZ3doV3VP?=
 =?utf-8?B?L0dnY2ZVL2ovSkNWWXNrWnk3bmQwYmZPUG9HQkNCR21qTEtZMktQclJwRWp4?=
 =?utf-8?B?M1oxUjZxdkVMWm9rQXhtYmJMdHA3Z0Q2R1J0bkxEaHEzdkRUMGZ5VEJXek12?=
 =?utf-8?B?d1FZcldJQ25Va1Z3amRnUURLRlBVMEwwblJ1dUgxRnFRWHdWTm5KOHc3SU5a?=
 =?utf-8?B?RDN6QUEyN0VNYWJnYnBLUVZ6UDgwMUNxTzlWbzMrVDQvVEdUZG5NdmZWK2N6?=
 =?utf-8?B?NnBCREpRdzRndm52VjQvc2hiUXRzcGdjL1ZGV1pCQ1FCWjFDckF5L3NSMXd6?=
 =?utf-8?B?cUE0VW8vck1Kc24rN01zR3NRMU9HaE9pTHNlclB3TFlqRS9jQjV6aHNFMFBO?=
 =?utf-8?B?Rk5qejhLeXNNdlgzeHFtZmpXMDB3MFRrTy82R2FZQ0tFWTNBK3pqSmU3VHJL?=
 =?utf-8?B?ZTdTZmlFSTI2K2JxdWFaMHVTUVpLSm91VnIzeStlR0xEVFJPemdEclNOQ1Nu?=
 =?utf-8?B?cm81cmdJejZTRWZIUVZjcTgzakFEbVdBVThqSlljQVhPT29GR2NmQXQ1RjF3?=
 =?utf-8?B?VXR4ZzZyUGJlZFg4dXJoeVpVOUZzM2V2TkE3L0VhdGpRMnVOVXB4K3pPczZM?=
 =?utf-8?B?SjJ0bVNPb3lGUjEzOWNJSVlqTHpmblREVHltaWg4bVJEZUNxbzNBV0Rla1Ny?=
 =?utf-8?B?NVp3dGRJYWg2aTBvUkJnWnBNbzU0VTVxaE9VRG82VjVDakxJUnoxbkZtQWxQ?=
 =?utf-8?B?UkFhb3dMVlZSSkVQdmhJN2pSRGM3WjZGVEVxdGNjZXhLazYrdjlRanNKbzVE?=
 =?utf-8?B?dGkzUjd6aDQxNTQ4VmEydjhxdjJyV0FNRzNHUGVZQlRTQmMxazlMVnBZbGNT?=
 =?utf-8?B?MGJON0ZscWZUbEVkOExGS1l3UDFteEJFeUFBU29Uak13cVJTM1c2NGVyMTZl?=
 =?utf-8?B?YnZZMUhpTHJRTGo5U0dUOWpQRDFNZDFvOHcyOTZuUi94OWhTVldQNzhJUHhl?=
 =?utf-8?B?UDBXZlB2dkN5QzAxQXp6RWZwcFdOUG9mS3Q2MTMwZUdlWGFYeUZ6OFFaYVlF?=
 =?utf-8?B?MWZVc3QxZWppelVPZ0ZBdkhpdjRZRWJRTlhCcS9UL2FoU2lGNDdzUnczcjJ2?=
 =?utf-8?B?Y2g5ZVhXN2V5eTlZRFlPSHV2RW5oc3lDamVXemg0RVVkMXJqblRjSDZheUN2?=
 =?utf-8?B?VWkzTW9IK2ZReG5DckxqcFRDT011UHI3cXhnTE1qQlhtaXlLazdXeDJyeXJ1?=
 =?utf-8?B?ZWFWYnlTSzBhZ1ZaMVpXczZmT2F6Y3lTRWlDbzRMYTY0WFQyQnNTTXRnWTRh?=
 =?utf-8?B?M3p5WmZsOUpCL0xVMGVDOWFHalQyQUZJdzh3VCtPZUxiaVgxUjVzbk1MOXJ6?=
 =?utf-8?B?dC90dlEwUitFQ0oxZGlFUURWandVdGpFSUs2d0lVbG55SDB3TkNER2hTSkV2?=
 =?utf-8?B?N2xIWG1oWUlzS1luQkNZeC9HN3h1RHNTM1J6MkNUTTBlcC9vbFhpUlJvTU1H?=
 =?utf-8?B?bTZxNDR1dktIUmhobXRFa1huK0NJZVpwVTF1OEZUWktEYWtlRzBlT3ZIbkkv?=
 =?utf-8?B?eUtHeVVSTXFVd1g3YkRvS1VOSG5JZ3N5ZWgzVmVvOUlJdTdFTWRIRTgrWVhY?=
 =?utf-8?Q?jzbz6rcgpPqkf5CxRlw+dsDms?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 78e01790-c64e-4eb1-dafd-08da958a1886
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB2900.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Sep 2022 13:15:58.1175
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: w4E1PSynndHYacbtKvTX5RykPxNyE+GVEcTW53p9wJnbWVVAl5KUVGzJSInzdfvrqnszfx9dUk+wj811olHgdw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5841
X-Spam-Status: No, score=-1.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        NICE_REPLY_A,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,
        SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 9/8/2022 2:35 AM, Bjorn Helgaas wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Fri, Aug 26, 2022 at 06:25:26PM +0530, Vidya Sagar wrote:
>> Previously ASPM L1 Substates control registers (CTL1 and CTL2) weren't
>> saved and restored during suspend/resume leading to L1 Substates
>> configuration being lost post-resume.
>>
>> Save the L1 Substates control registers so that the configuration is
>> retained post-resume.
>>
>> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
>> ---
>> V3:
>> * Disabled L1.2 enable fields while restoring Control-1 register
> 
> This really looks promising!  Has somebody confirmed that the
> disappearing L1SS capability problem doesn't happen here?

Based on the update from Lukasz in 
https://patchwork.kernel.org/project/linux-pci/patch/20220705060014.10050-1-vidyas@nvidia.com/ 
(last update), the L1SS capability registers disappearing issue is not 
due to this change and they are working on it seems.

> 
>> +void pci_save_aspm_l1ss_state(struct pci_dev *dev)
>> +{
>> +     int aspm_l1ss;
>> +     struct pci_cap_saved_state *save_state;
>> +     u32 *cap;
>> +
>> +     if (!pci_is_pcie(dev))
>> +             return;
>> +
>> +     aspm_l1ss = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_L1SS);
>> +     if (!aspm_l1ss)
>> +             return;
> 
> Isn't it enough to check this?

Yup. I'll address it in the next version.

> 
>    if (!dev->l1ss)
>      return;
> 
>> +void pci_restore_aspm_l1ss_state(struct pci_dev *dev)
>> +{
>> +     int aspm_l1ss;
>> +     struct pci_cap_saved_state *save_state;
>> +     u32 *cap, l1_2_enable;
>> +
>> +     if (!pci_is_pcie(dev))
>> +             return;
>> +
>> +     aspm_l1ss = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_L1SS);
>> +     if (!aspm_l1ss)
>> +             return;
>> +
>> +     save_state = pci_find_saved_ext_cap(dev, PCI_EXT_CAP_ID_L1SS);
>> +     if (!save_state)
>> +             return;
>> +
>> +     cap = (u32 *)&save_state->cap.data[0];
>> +     pci_write_config_dword(dev, aspm_l1ss + PCI_L1SS_CTL2, *cap++);
>> +     /* Disable L1.2 while updating.  See PCIe r5.0, sec 5.5.4, 7.8.3.3 */
>> +     l1_2_enable = *cap & PCI_L1SS_CTL1_L1_2_MASK;
>> +     pci_write_config_dword(dev, aspm_l1ss + PCI_L1SS_CTL1,
>> +                            (*cap & ~PCI_L1SS_CTL1_L1_2_MASK));
>> +     if (l1_2_enable)
>> +             pci_clear_and_set_dword(dev, aspm_l1ss + PCI_L1SS_CTL1, 0,
>> +                                     l1_2_enable);
>> +}
> 
> What if we did something like the following?  Then we wouldn't have to
> duplicate the fancy logic in aspm_calc_l1ss_info() and
> pci_restore_aspm_l1ss_state(), and we'd only need the big comment in
> one place.

I'll refactor the existing code accommodating this suggestion and push 
L1SS save/restore change on top of it in the next version.

Thanks for your review comments.

> 
> +static void aspm_program_l1ss(struct pci_dev *dev, u32 ctl1, u32 ctl2)
> +{
> +       u16 l1ss = dev->l1ss;
> +       u32 l1_2_enable;
> +
> +       /*
> +        * Per PCIe r6.0, sec 5.5.4, T_POWER_ON in PCI_L1SS_CTL2 must be
> +        * programmed prior to setting the L1.2 enable bits in PCI_L1SS_CTL1.
> +        */
> +       pci_write_config_dword(dev, l1ss + PCI_L1SS_CTL2, ctl2);
> +
> +       /*
> +        * In addition, Common_Mode_Restore_Time and LTR_L1.2_THRESHOLD in
> +        * PCI_L1SS_CTL1 must be programmed *before* setting the L1.2
> +        * enable bits, even though they're all in PCI_L1SS_CTL1.
> +        */
> +       l1_2_enable = ctl1 & PCI_L1SS_CTL1_L1_2_MASK;
> +       ctl1 &= ~PCI_L1SS_CTL1_L1_2_MASK;
> +
> +       pci_write_config_dword(dev, l1ss + PCI_L1SS_CTL1, ctl1);
> +       if (l1_2_enable)
> +               pci_write_config_dword(dev, l1ss + PCI_L1SS_CTL1,
> +                                      ctl1 | l1_2_enable);
> +}
> 
> (This is somewhat simplified from what aspm_calc_l1ss_info() does
> today.  It looks to me like aspm_calc_l1ss_info() does more config
> reads than necessary.)
>