Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 14 Apr 2022 11:41:34 -0500
From:   Bjorn Helgaas <helgaas@kernel.org>
To:     Kai-Heng Feng <kai.heng.feng@canonical.com>
Cc:     Vidya Sagar <vidyas@nvidia.com>,
        "Kenneth R. Crudup" <kenny@panix.com>, bhelgaas@google.com,
        lorenzo.pieralisi@arm.com, hkallweit1@gmail.com,
        wangxiongfeng2@huawei.com, mika.westerberg@linux.intel.com,
        chris.packham@alliedtelesis.co.nz, yangyicong@hisilicon.com,
        treding@nvidia.com, jonathanh@nvidia.com, abhsahu@nvidia.com,
        sagupta@nvidia.com, linux-pci@vger.kernel.org,
        linux-kernel@vger.kernel.org, kthota@nvidia.com,
        mmaddireddy@nvidia.com, sagar.tv@gmail.com,
        Ricky Wu <ricky_wu@realtek.com>,
        Rajat Jain <rajatja@google.com>,
        Prasad Malisetty <quic_pmaliset@quicinc.com>,
        Victor Ding <victording@google.com>
Subject: Re: [PATCH V1] PCI/ASPM: Save/restore L1SS Capability for
 suspend/resume
Message-ID: <20220414164134.GA751756@bhelgaas>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAAd53p4r1v_sN=8kv_fOx_VEb3k=4rU9iw52LfmEHO1crnms=g@mail.gmail.com>
Precedence: bulk

On Wed, Apr 13, 2022 at 08:19:26AM +0800, Kai-Heng Feng wrote:
> On Wed, Apr 13, 2022 at 6:50 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > ...

> >   - For L1 PM Substates configuration, sec 5.5.4 says that both ports
> >     must be configured while ASPM L1 is disabled, but I don't think we
> >     currently guarantee this: we restore all the upstream component
> >     state first, and we don't know the ASPM state of the downstream
> >     one.  Maybe we need to:
> >
> >       * When restoring upstream component,
> >           + disable its ASPM
> >
> >       * When restoring downstream component,
> >           + disable its ASPM
> >           + restore upstream component's LTR, L1SS
> >           + restore downstream component's LTR, L1SS
> >           + restore upstream component's ASPM
> >           + restore downstream component's ASPM
> 
> Right now L1SS isn't reprogrammed after S3, and that causes WD NVMe
> starts to spew lots of AER errors.

Right now we don't fully restore L1SS-related state after S3, so maybe
there's some inconsistency that leads to the AER errors.

Could you collect the "lspci -vv" state before and after S3 so we can
compare them?

> So yes please restore L1SS upon resume. Meanwhile I am asking vendor
> _why_ restoring L1SS is crucial for it to work.
> 
> I also wonder what's the purpose of pcie_aspm_pm_state_change()? Can't
> we just restore ASPM bits like other configs?

Good question.  What's the context?  This is in the
pci_raw_set_power_state() path, not the pci_restore_state() path, so
seems like a separate discussion.

Bjorn