Received-SPF: pass (google.com: domain of linux-kernel+bounces-56747-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <0e7944d410664153b506ea584d92cd6bb0a93f6a.camel@linux.intel.com>
Subject: Re: [PATCH v3 3/3] PCI/ASPM: Fix L1SS parameters & only enable
 supported features when enable link state
From: "David E. Box" <david.e.box@linux.intel.com>
Reply-To: david.e.box@linux.intel.com
To: Jian-Hong Pan <jhp@endlessos.org>, Bjorn Helgaas <helgaas@kernel.org>, 
 Johan Hovold <johan@kernel.org>, Ilpo =?ISO-8859-1?Q?J=E4rvinen?=
 <ilpo.jarvinen@linux.intel.com>
Cc: Mika Westerberg <mika.westerberg@linux.intel.com>, Nirmal Patel
 <nirmal.patel@linux.intel.com>, Jonathan Derrick
 <jonathan.derrick@linux.dev>,  linux-pci@vger.kernel.org,
 linux-kernel@vger.kernel.org, linux@endlessos.org
Date: Wed, 07 Feb 2024 08:02:48 -0800
In-Reply-To: <20240207111854.576402-2-jhp@endlessos.org>
References: <20240207111854.576402-2-jhp@endlessos.org>
Organization: David E. Box
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.44.4-0ubuntu2 
Precedence: bulk
MIME-Version: 1.0

On Wed, 2024-02-07 at 19:18 +0800, Jian-Hong Pan wrote:
> The original __pci_enable_link_state() configs the links directly without=
:
> * Check the L1 substates features which are supported, or not
> * Calculate & program related parameters for L1.2, such as T_POWER_ON,
> =C2=A0 Common_Mode_Restore_Time, and LTR_L1.2_THRESHOLD
>=20
> This leads some supported L1 PM substates of the link between VMD remappe=
d
> PCIe Root Port and NVMe get wrong configs when a caller tries to enabled
> them.
>=20
> Here is a failed example on ASUS B1400CEAE with enabled VMD:
>=20
> Capabilities: [900 v1] L1 PM Substates
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 L1SubCap: PCI-PM_L1.2+ PCI-PM_=
L1.1- ASPM_L1.2+ ASPM_L1.1-
> L1_PM_Substates+
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 PortCommonModeRestoreTime=3D32us PortTPowerO=
nTime=3D10us
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 L1SubCtl1: PCI-PM_L1.2- PCI-PM=
_L1.1- ASPM_L1.2+ ASPM_L1.1-
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 T_CommonMode=3D0us LTR1.2_Threshold=3D=
0ns
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 L1SubCtl2: T_PwrOn=3D10us
>=20
> This patch initializes the link's L1 PM substates to get the supported
> features and programs relating paramters, if some of them are going to be
> enabled in __pci_enable_link_state(). Then, enables the L1 PM substates i=
f
> the caller intends to enable them and they are supported.
>=20
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=3D218394
> Signed-off-by: Jian-Hong Pan <jhp@endlessos.org>
> ---
> v2:
> - Prepare the PCIe LTR parameters before enable L1 Substates
>=20
> v3:
> - Only enable supported features for the L1 Substates part
>=20
> =C2=A0drivers/pci/pcie/aspm.c | 12 +++++++-----
> =C2=A01 file changed, 7 insertions(+), 5 deletions(-)
>=20
> diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
> index a39d2ee744cb..c866971cae70 100644
> --- a/drivers/pci/pcie/aspm.c
> +++ b/drivers/pci/pcie/aspm.c
> @@ -1389,14 +1389,16 @@ static int __pci_enable_link_state(struct pci_dev
> *pdev, int state, bool locked)
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0link->aspm_default |=3D ASPM_STATE_L0S;
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if (state & PCIE_LINK_STA=
TE_L1)
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0link->aspm_default |=3D ASPM_STATE_L1;
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0/* L1 PM substates require L1 =
*/
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if (state & PCIE_LINK_STATE_L1=
_1)
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if (state & ASPM_STATE_L1_2_MA=
SK)
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0aspm_l1ss_init(link);

This mixes ASPM_STATE flags with PCIE_LINK_STATE register mapping. This may=
 work
but I don't know if it's intended to. Rather do,

    if (link->default & ASPM_STATE_L1_2_MASK)

after collecting all of the states to be enabled.

I understand that you are calling aspm_l1ss_init() to do the L1.2 calculati=
ons
but it does more than this that you don't need. Maybe it would be more
appropriate to call aspm_calc_l12_info() directly through an additional fun=
ction
that finds the parent and determines both ends of the link support L1SS.

> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0/* L1 PM substates require L1 =
and should be in supported list */
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if (state & link->aspm_support=
 & PCIE_LINK_STATE_L1_1)
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0link->aspm_default |=3D ASPM_STATE_L1_1 | ASPM_STAT=
E_L1;
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if (state & PCIE_LINK_STATE_L1=
_2)
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if (state & link->aspm_support=
 & PCIE_LINK_STATE_L1_2)
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0link->aspm_default |=3D ASPM_STATE_L1_2 | ASPM_STAT=
E_L1;
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if (state & PCIE_LINK_STATE_L1=
_1_PCIPM)
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if (state & link->aspm_support=
 & PCIE_LINK_STATE_L1_1_PCIPM)
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0link->aspm_default |=3D ASPM_STATE_L1_1_PCIPM | ASP=
M_STATE_L1;
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if (state & PCIE_LINK_STATE_L1=
_2_PCIPM)
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if (state & link->aspm_support=
 & PCIE_LINK_STATE_L1_2_PCIPM)
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0link->aspm_default |=3D ASPM_STATE_L1_2_PCIPM | ASP=
M_STATE_L1;
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0pcie_config_aspm_link(lin=
k, policy_to_aspm_state(link));
> =C2=A0

I don't think these changes are necessary. pcie_config_aspm_link() already
checks link->aspm_capable which was initialized from link->aspm_support.

David