Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <56608a4d-aa7e-70d0-e960-943ea9e70525@nvidia.com>
Date:   Sun, 9 Oct 2022 11:14:30 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.3.2
Subject: Re: [PATCH V4 1/2] PCI/ASPM: Refactor ASPM L1SS control register
 programming
Content-Language: en-US
To:     Bjorn Helgaas <helgaas@kernel.org>
Cc:     bhelgaas@google.com, lorenzo.pieralisi@arm.com,
        refactormyself@gmail.com, kw@linux.com, rajatja@google.com,
        kenny@panix.com, kai.heng.feng@canonical.com, abhsahu@nvidia.com,
        sagupta@nvidia.com, treding@nvidia.com, jonathanh@nvidia.com,
        linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org,
        kthota@nvidia.com, mmaddireddy@nvidia.com, sagar.tv@gmail.com,
        Krishna chaitanya chundru <quic_krichai@quicinc.com>
References: <20221005023542.GA2190062@bhelgaas>
From:   Vidya Sagar <vidyas@nvidia.com>
In-Reply-To: <20221005023542.GA2190062@bhelgaas>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NHRnTXZUMHZHYVdMUlFQSnBmT1ZwQjZrSjVWWlY1a0VjRlRyb3IxMWpQK3ZQ?=
 =?utf-8?B?Wkt1RWk0UW1yNGdyaURzWFA4bUxyUWU4VjJhWTk2Wk8rNGNLRkJRWjd1TkhR?=
 =?utf-8?B?eG9oeGRIVmRqdUlabWJTazlUeFl5TGVIbW1LU0FwQjdBWjEvZlZEdEU1MzMw?=
 =?utf-8?B?OURGMWo3NWRMNTJkdDMyb2pKYmFVMGFpTmVZajNlK0RZOEZwM3g0UzRsTmVJ?=
 =?utf-8?B?ektiRXAvd3NWOVFNR3hwS3JMejl0cXRsd3pnVnB0SWNOcmdGMkt2RDJKUUMv?=
 =?utf-8?B?V0lORVZielM1UFljSDl0elZBVWQ3UzJFZzhnVi9pS2xtSEtuT3NQdjZBR2cr?=
 =?utf-8?B?QzFrSzduUWRpQkhyaDFJbW0zUHNSWVF0cnRuNVE1WE5zUmRxN3hpb01mNzdk?=
 =?utf-8?B?eThvK0w2YWV3MHFCM0lEbWNWZnZwSkZHTG9sREtCV1dJL1VwNjBaVHV2Zkpn?=
 =?utf-8?B?QUtZSXFUWWtMdHEveWpZRWl2cFpJcnNSQlcyRDBsL2xOTnBNN00xOTVnY3J3?=
 =?utf-8?B?SXlicGRGdVJVSXZKVUxKZWRCckwxVTJOK2RNcUxMN3plbXE0QU1XZDdnZXIz?=
 =?utf-8?B?WC8zU0lMUHBFcXZyRUpDU282UjV0eFM2SjkvT0JtUm41dUJ5ZFl0L0c4dTN1?=
 =?utf-8?B?VE4yVGpiYlFEVmI0ajRMOXBCM1VlUEdDMGlqR3JINlhjYU94UnRGYlJidUhG?=
 =?utf-8?B?dzArWUFKZEMzTHM0aFBENm1CRW9zdU5mYmJ2NDhOK1FrZHZ6dW9GUmtueTVl?=
 =?utf-8?B?ckE2L29aUGJ6YlhEOHFzNURkTlBER3ZJWUxTbUpHWmQyNzAwSUx3aDBlUnpV?=
 =?utf-8?B?NnliK1kra1lNWUt5UXZCUjN5ZUNCdlY4RDFKazROTnB1cm9tUzMrZHFXN1FI?=
 =?utf-8?B?MWRQU0NGWWR5OGNHZXdnc2k5V3RyN2hsbFFqdCtOcm5sOUhZd3k5amZNY24x?=
 =?utf-8?B?NmtUT0pnY1l1NVluTEFDS2JoSVMxcm5ySm85L0xIVFVoNWFidnVkMkdQQUJr?=
 =?utf-8?B?R043MXdIZTIzdUdsQkRQY0x2dS9pa21KcnVwVi9Jb3lPcGE0MjBtOUV3R2Yv?=
 =?utf-8?B?S1pPK25LS3lUN0twNHRiaDNuTU5ZelNoYWV2VEVCOGl5WXNXV2dtN3NkbFpS?=
 =?utf-8?B?cEFxQzZHbU5WcGJRL3RBa25ZVWxIRTVTTTVDcWJaZGl0dlpYNlVjZGY2UjQ1?=
 =?utf-8?B?OGlXU1R1UzNIMkNEOHNOMkcxcVNsU3lNdmtubTJMRDBaN2Fjc1BLSGVLOVFs?=
 =?utf-8?B?NU94ZXM0cUNDK0xCVW1pU3BETUZQTjdJaHFDVHJVNnhHR2EvVFlVK0hOajhT?=
 =?utf-8?B?b2xraVM1a2ZxSGZlWjRqZ1hKTFRyWlJqMkxRTVRnMWRDbmlPaXZSWk56eHNt?=
 =?utf-8?B?a1hhQTE4TXpuM2oxQ3RjenhHM1hRdzBJVzNla1Q3ek4yRS9QZEIycXJ2MXJ3?=
 =?utf-8?B?MmFxeFRucVNFVTdoeURTMjgvSHRyZE1xTnBIZFJaWTRTbUNaTWExVDdmcERj?=
 =?utf-8?B?QkVQaVVYaldZUk5mUUp6TUtyMWhrNTJKYmFGbWx3TVNSYmNxSm4zNHB4MWdX?=
 =?utf-8?B?cmdDOUIydXNUc2kyeVdFTndTT1FmdTJSNWNWU0FFcGs3a0E2bDZDcmJXSFZP?=
 =?utf-8?B?OENXMXJuZm4xUXhjdHRIZmQ3QzZaK2svNVNWRGtSQ0M5MW5DR0pLcDlTUUZJ?=
 =?utf-8?B?MW43dzY4Q0dPMDZ6V0YvQ0RNUmx5eTZTNFBVcksvQldSY0hMUmgwQjgzdDJT?=
 =?utf-8?B?WWQ2S2NQcjZxdTNjK0JoS25WL2lFZFpjSmhLQ1dLdkZ1VXJBNVYvRDJCNWdv?=
 =?utf-8?B?ejZMTm5qNzNQeG5taW9HRDFhanlsNll1YXBvS3NyaXVlSXFYVTROeEpzam9q?=
 =?utf-8?B?dENSRFB3aTRBVjBvTER0L3VkZGJUM3Q1bjk2Yy9MdmgyWFIzRllqKzBPTGc5?=
 =?utf-8?B?SVdXeE1obTBDLzkzR0pWdGl5SmxzK2NSSHJnSmRPTmdTS0hnRTlwVnhaWTFW?=
 =?utf-8?B?b09qVlovRjdCVURuck5wUnVNMFVjaXJvdjdiY0htUnBVQVBReE5NQS92anQ2?=
 =?utf-8?B?MG1uU3FzaXUyeXhySXJmR2R2TUYvT3ZyM3AxMXBIc0duVmphNDNrRG9telo0?=
 =?utf-8?Q?lDBzyGt2DedK4gbotKIvSBByz?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 52d7a548-45a3-4986-e104-08daa9b95cd4
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB2900.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2022 05:44:42.1751
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: xJnmtGpprb0yUE4JHSOS5mLru1vSmvNXzb1EvsPsNDG0nJYMJ/6uuy+xViwY0y7HLctmNHMl2gpaAxSrVdvh9Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8PR12MB5462
X-Spam-Status: No, score=-4.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        NICE_REPLY_A,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,
        SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 10/5/2022 8:05 AM, Bjorn Helgaas wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Mon, Oct 03, 2022 at 12:12:22PM +0530, Vidya Sagar wrote:
>> On 9/30/2022 3:30 AM, Bjorn Helgaas wrote:
>>> On Tue, Sep 13, 2022 at 06:48:21PM +0530, Vidya Sagar wrote:
>>>> Refactor the code to extract the command code out to program
>>>> Control Registers-1 & 2 of L1 Sub-States capability to a new function
>>>> aspm_program_l1ss() and call it for both parent and child devices.
>>>>
>>>> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
>>>> ---
>>>> V4:
>>>> * New patch in this series
>>>>
>>>>    drivers/pci/pcie/aspm.c | 63 +++++++++++++++++++----------------------
>>>>    1 file changed, 29 insertions(+), 34 deletions(-)
>>>>
>>>> diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
>>>> index a8aec190986c..ecbe3af4188d 100644
>>>> --- a/drivers/pci/pcie/aspm.c
>>>> +++ b/drivers/pci/pcie/aspm.c
>>>> @@ -455,6 +455,31 @@ static void pci_clear_and_set_dword(struct pci_dev *pdev, int pos,
>>>>         pci_write_config_dword(pdev, pos, val);
>>>>    }
>>>>
>>>> +static void aspm_program_l1ss(struct pci_dev *dev, u32 ctl1, u32 ctl2)
>>>> +{
>>>> +     u16 l1ss = dev->l1ss;
>>>> +     u32 l1_2_enable;
>>>> +
>>>> +     /*
>>>> +      * Per PCIe r6.0, sec 5.5.4, T_POWER_ON in PCI_L1SS_CTL2 must be
>>>> +      * programmed prior to setting the L1.2 enable bits in PCI_L1SS_CTL1.
>>>> +      */
>>>> +     pci_write_config_dword(dev, l1ss + PCI_L1SS_CTL2, ctl2);
>>>> +
>>>> +     /*
>>>> +      * In addition, Common_Mode_Restore_Time and LTR_L1.2_THRESHOLD in
>>>> +      * PCI_L1SS_CTL1 must be programmed *before* setting the L1.2
>>>> +      * enable bits, even though they're all in PCI_L1SS_CTL1.
>>>> +      */
>>>> +     l1_2_enable = ctl1 & PCI_L1SS_CTL1_L1_2_MASK;
>>>> +     ctl1 &= ~PCI_L1SS_CTL1_L1_2_MASK;
>>>> +
>>>> +     pci_write_config_dword(dev, l1ss + PCI_L1SS_CTL1, ctl1);
>>>> +     if (l1_2_enable)
>>>> +             pci_write_config_dword(dev, l1ss + PCI_L1SS_CTL1,
>>>> +                                    ctl1 | l1_2_enable);
>>>> +}
>>>> +
>>>>    /* Calculate L1.2 PM substate timing parameters */
>>>>    static void aspm_calc_l1ss_info(struct pcie_link_state *link,
>>>>                                 u32 parent_l1ss_cap, u32 child_l1ss_cap)
>>>> @@ -464,7 +489,6 @@ static void aspm_calc_l1ss_info(struct pcie_link_state *link,
>>>>         u32 t_common_mode, t_power_on, l1_2_threshold, scale, value;
>>>>         u32 ctl1 = 0, ctl2 = 0;
>>>>         u32 pctl1, pctl2, cctl1, cctl2;
>>>> -     u32 pl1_2_enables, cl1_2_enables;
>>>>
>>>>         if (!(link->aspm_support & ASPM_STATE_L1_2_MASK))
>>>>                 return;
>>>> @@ -513,39 +537,10 @@ static void aspm_calc_l1ss_info(struct pcie_link_state *link,
>>>>             ctl2 == pctl2 && ctl2 == cctl2)
>>>>                 return;
>>>>
>>>> -     /* Disable L1.2 while updating.  See PCIe r5.0, sec 5.5.4, 7.8.3.3 */
>>>> -     pl1_2_enables = pctl1 & PCI_L1SS_CTL1_L1_2_MASK;
>>>> -     cl1_2_enables = cctl1 & PCI_L1SS_CTL1_L1_2_MASK;
>>>> -
>>>> -     if (pl1_2_enables || cl1_2_enables) {
>>>> -             pci_clear_and_set_dword(child, child->l1ss + PCI_L1SS_CTL1,
>>>> -                                     PCI_L1SS_CTL1_L1_2_MASK, 0);
>>>> -             pci_clear_and_set_dword(parent, parent->l1ss + PCI_L1SS_CTL1,
>>>> -                                     PCI_L1SS_CTL1_L1_2_MASK, 0);
>>>> -     }
>>>> -
>>>> -     /* Program T_POWER_ON times in both ports */
>>>> -     pci_write_config_dword(parent, parent->l1ss + PCI_L1SS_CTL2, ctl2);
>>>> -     pci_write_config_dword(child, child->l1ss + PCI_L1SS_CTL2, ctl2);
>>>> -
>>>> -     /* Program Common_Mode_Restore_Time in upstream device */
>>>> -     pci_clear_and_set_dword(parent, parent->l1ss + PCI_L1SS_CTL1,
>>>> -                             PCI_L1SS_CTL1_CM_RESTORE_TIME, ctl1);
>>>> -
>>>> -     /* Program LTR_L1.2_THRESHOLD time in both ports */
>>>> -     pci_clear_and_set_dword(parent, parent->l1ss + PCI_L1SS_CTL1,
>>>> -                             PCI_L1SS_CTL1_LTR_L12_TH_VALUE |
>>>> -                             PCI_L1SS_CTL1_LTR_L12_TH_SCALE, ctl1);
>>>> -     pci_clear_and_set_dword(child, child->l1ss + PCI_L1SS_CTL1,
>>>> -                             PCI_L1SS_CTL1_LTR_L12_TH_VALUE |
>>>> -                             PCI_L1SS_CTL1_LTR_L12_TH_SCALE, ctl1);
>>>> -
>>>> -     if (pl1_2_enables || cl1_2_enables) {
>>>> -             pci_clear_and_set_dword(parent, parent->l1ss + PCI_L1SS_CTL1, 0,
>>>> -                                     pl1_2_enables);
>>>> -             pci_clear_and_set_dword(child, child->l1ss + PCI_L1SS_CTL1, 0,
>>>> -                                     cl1_2_enables);
>>>> -     }
>>>> +     aspm_program_l1ss(parent,
>>>> +                       ctl1 | (pctl1 & PCI_L1SS_CTL1_L1_2_MASK), ctl2);
>>>> +     aspm_program_l1ss(child,
>>>> +                       ctl1 | (cctl1 & PCI_L1SS_CTL1_L1_2_MASK), ctl2);
>>>
>>> This doesn't seem right to me.  I think the intent is to update
>>> LTR_L1.2_THRESHOLD and Common_Mode_Restore_Time, which are encoded in
>>> "ctl1".  It does do that, but it looks like it *also* clears
>>> everything except PCI_L1SS_CTL1_L1_2_MASK, i.e., the L1.1 Enable bits,
>>> the Link Activation bits, and the RsvdP bits, which I don't think we
>>> should be clearing.  Am I missing something?
>>
>> Agree. Instead of updating some of the register fields with new values while
>> keeping other fields intact, this code programs the register fields with
>> only new values and also programming all other fields to zero which is
>> wrong.
>> Thanks for catching this. I missed it as the card with which I had tested
>> didn't have L1.1 support.
> 
>> I think the following modification should fix this issue.
>>
>>
>> @@ -537,10 +537,23 @@ static void aspm_calc_l1ss_info(struct pcie_link_state
>> *link,
>>              ctl2 == pctl2 && ctl2 == cctl2)
>>                  return;
>>
>> -       aspm_program_l1ss(parent,
>> -                         ctl1 | (pctl1 & PCI_L1SS_CTL1_L1_2_MASK), ctl2);
>> -       aspm_program_l1ss(child,
>> -                         ctl1 | (cctl1 & PCI_L1SS_CTL1_L1_2_MASK), ctl2);
>> +       pctl1 &= ~(PCI_L1SS_CTL1_CM_RESTORE_TIME |
>> +                  PCI_L1SS_CTL1_LTR_L12_TH_VALUE |
>> +                  PCI_L1SS_CTL1_LTR_L12_TH_SCALE);
>> +       pctl1 |= (ctl1 & (PCI_L1SS_CTL1_CM_RESTORE_TIME |
>> +                         PCI_L1SS_CTL1_LTR_L12_TH_VALUE |
>> +                         PCI_L1SS_CTL1_LTR_L12_TH_SCALE)
>> +                );
>> +       aspm_program_l1ss(parent, pctl1, ctl2);
>> +
>> +       cctl1 &= ~(PCI_L1SS_CTL1_CM_RESTORE_TIME |
>> +                  PCI_L1SS_CTL1_LTR_L12_TH_VALUE |
>> +                  PCI_L1SS_CTL1_LTR_L12_TH_SCALE);
>> +       cctl1 |= (ctl1 & (PCI_L1SS_CTL1_CM_RESTORE_TIME |
>> +                         PCI_L1SS_CTL1_LTR_L12_TH_VALUE |
>> +                         PCI_L1SS_CTL1_LTR_L12_TH_SCALE)
>> +                );
>> +       aspm_program_l1ss(child, cctl1, ctl2);
>>   }
>>
>>   static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist)
>>
>> But, given that you mentioned we shouldn't be touching Rsvd also, I'm
>> wondering if the following can cause any issue?
>> With the top of the tree code, the CMRT in ctrl1 register is updated only
>> for the upstream device whereas with my change, it gets updated even for the
>> downstream device. Although spec says that it is Rsvd for a downstream
>> device (i.e. upstream port), I'm wondering if we should really avoid
>> touching it?
> 
> I'm not sure it's worth bothering about at this point.  It feels a
> little OCD right now.
> 
>> If the answer is yes, I think it is better to drop the modifications done to
>> aspm_calc_l1ss_info() function and just proceed with rest of the
>> modifications given the way it is differentiating between upstream and
>> downstream devices while updating the registers.
>> What are your comments on this?
> 
> I thought we had some indication that Common_Mode_Restore_Time and
> LTR_L1.2_THRESHOLD should be programmed *before* setting the L1.2
> enable bits, even though they're all in PCI_L1SS_CTL1.
> 
> The first place that comment appears is
> https://lore.kernel.org/linux-pci/20220907210540.GA140988@bhelgaas/,
> but I can't remember if it actually solved a problem or if it was just
> more OCD reading of the spec, which says "This field must only be
> modified when the ASPM L1.2 Enable bit is Clear".
> 
> I'm inclined to keep the aspm_program_l1ss() changes unless we think
> they're risky, because I think it is a small step forward, at least in
> terms of reducing the number of config accesses.
> 
> On pci/aspm, I currently have your v4 patches plus these tweaks:

Thanks Bjorn for pushing these modifications and I'm fine with the below 
changes.

> 
> diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
> index dc2e21c7a9d4..016d222b07c7 100644
> --- a/drivers/pci/pcie/aspm.c
> +++ b/drivers/pci/pcie/aspm.c
> @@ -537,10 +537,21 @@ static void aspm_calc_l1ss_info(struct pcie_link_state *link,
>              ctl2 == pctl2 && ctl2 == cctl2)
>                  return;
> 
> -       aspm_program_l1ss(parent,
> -                         ctl1 | (pctl1 & PCI_L1SS_CTL1_L1_2_MASK), ctl2);
> -       aspm_program_l1ss(child,
> -                         ctl1 | (cctl1 & PCI_L1SS_CTL1_L1_2_MASK), ctl2);
> +       pctl1 &= ~(PCI_L1SS_CTL1_CM_RESTORE_TIME |
> +                  PCI_L1SS_CTL1_LTR_L12_TH_VALUE |
> +                  PCI_L1SS_CTL1_LTR_L12_TH_SCALE);
> +       pctl1 |= (ctl1 & (PCI_L1SS_CTL1_CM_RESTORE_TIME |
> +                         PCI_L1SS_CTL1_LTR_L12_TH_VALUE |
> +                         PCI_L1SS_CTL1_LTR_L12_TH_SCALE));
> +       aspm_program_l1ss(parent, pctl1, ctl2);
> +
> +       cctl1 &= ~(PCI_L1SS_CTL1_CM_RESTORE_TIME |
> +                  PCI_L1SS_CTL1_LTR_L12_TH_VALUE |
> +                  PCI_L1SS_CTL1_LTR_L12_TH_SCALE);
> +       cctl1 |= (ctl1 & (PCI_L1SS_CTL1_CM_RESTORE_TIME |
> +                         PCI_L1SS_CTL1_LTR_L12_TH_VALUE |
> +                         PCI_L1SS_CTL1_LTR_L12_TH_SCALE));
> +       aspm_program_l1ss(child, cctl1, ctl2);
>   }
> 
>   static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist)
> @@ -727,9 +738,6 @@ void pci_save_aspm_l1ss_state(struct pci_dev *dev)
>          u16 l1ss = dev->l1ss;
>          u32 *cap;
> 
> -       if (!pci_is_pcie(dev))
> -               return;
> -
>          if (!l1ss)
>                  return;
> 
> @@ -748,9 +756,6 @@ void pci_restore_aspm_l1ss_state(struct pci_dev *dev)
>          u32 *cap, ctl1, ctl2;
>          u16 l1ss = dev->l1ss;
> 
> -       if (!pci_is_pcie(dev))
> -               return;
> -
>          if (!l1ss)
>                  return;
>