Received-SPF: pass (google.com: domain of linux-kernel+bounces-159181-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Thu, 25 Apr 2024 14:01:44 -0700
From: Nirmal Patel <nirmal.patel@linux.intel.com>
To: Jian-Hong Pan <jhp@endlessos.org>
Cc: Bjorn Helgaas <helgaas@kernel.org>, Johan Hovold <johan@kernel.org>,
 David Box <david.e.box@linux.intel.com>, Ilpo =?ISO-8859-1?Q?J=E4rvinen?=
 <ilpo.jarvinen@linux.intel.com>, Kuppuswamy Sathyanarayanan
 <sathyanarayanan.kuppuswamy@linux.intel.com>, Mika Westerberg
 <mika.westerberg@linux.intel.com>, Damien Le Moal <dlemoal@kernel.org>,
 Jonathan Derrick <jonathan.derrick@linux.dev>, linux-pci@vger.kernel.org,
 linux-kernel@vger.kernel.org, <paul.m.stillwell.jr@intel.com>
Subject: Re: [PATCH v5 0/4] PCI: vmd: Enable PCI PM's L1 substates of
 remapped PCIe Root Port and NVMe
Message-ID: <20240425140144.000049a4@linux.intel.com>
In-Reply-To: <20240424105814.21690-2-jhp@endlessos.org>
References: <20240424105814.21690-2-jhp@endlessos.org>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

On Wed, 24 Apr 2024 18:58:15 +0800
Jian-Hong Pan <jhp@endlessos.org> wrote:

> Re-send for the version information.
> 
> Notice the VMD remapped PCIe Root Port and NVMe have PCI PM L1
> substates capability, but they are disabled originally.
> 
> Here is a failed example on ASUS B1400CEAE with enabled VMD:
> 
> 10000:e0:06.0 PCI bridge: Intel Corporation 11th Gen Core Processor
> PCIe Controller (rev 01) (prog-if 00 [Normal decode]) ...
>     Capabilities: [200 v1] L1 PM Substates
>         L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
> L1_PM_Substates+ PortCommonModeRestoreTime=45us PortTPowerOnTime=50us
>         L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1-
>         	   T_CommonMode=45us LTR1.2_Threshold=101376ns
>         L1SubCtl2: T_PwrOn=50us
> 
> 10000:e1:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue
> SN550 NVMe SSD (rev 01) (prog-if 02 [NVM Express]) ...
>     Capabilities: [900 v1] L1 PM Substates
>         L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1-
> L1_PM_Substates+ PortCommonModeRestoreTime=32us PortTPowerOnTime=10us
>         L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1-
>                    T_CommonMode=0us LTR1.2_Threshold=0ns
>         L1SubCtl2: T_PwrOn=10us
> 
> According to "PCIe r6.0, sec 5.5.4", to config the link between the
> PCIe Root Port and the child device correctly:
> * Ensure both devices are in D0 before enabling PCI-PM L1 PM
> Substates.
> * Ensure L1.2 parameters: Common_Mode_Restore_Times, T_POWER_ON and
>   LTR_L1.2_THRESHOLD are programmed properly on both devices before
> enable bits for L1.2.
> 
> Prepare this series to fix that.
> 
> Jian-Hong Pan (4):
>   PCI: vmd: Set PCI devices to D0 before enable PCI PM's L1 substates
>   PCI/ASPM: Add notes about enabling PCI-PM L1SS to
> pci_enable_link_state(_locked) PCI/ASPM: Introduce aspm_get_l1ss_cap()
>   PCI/ASPM: Fix L1.2 parameters when enable link state
> 
>  drivers/pci/controller/vmd.c | 13 ++++++++----
>  drivers/pci/pcie/aspm.c      | 41
> ++++++++++++++++++++++++++++-------- 2 files changed, 41
> insertions(+), 13 deletions(-)
> 

Hi,

We are running some tests to make sure we dont have issue with other
platforms and trying to avoid another hotplug scenario. Please wait for
our Ack before merging this patch. Thanks.

-nirmal