Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Message-ID: <f02332767323fc3ecccea13dd47ecfff12526112.camel@ettle.org.uk>
Subject: Re: rtsx_pci not restoring ASPM state after suspend/resume
From:   James Ettle <james@ettle.org.uk>
To:     Bjorn Helgaas <helgaas@kernel.org>
Cc:     =?UTF-8?Q?=E5=90=B3=E6=98=8A=E6=BE=84?= Ricky 
        <ricky_wu@realtek.com>, Rui Feng <rui_feng@realsil.com.cn>,
        Arnd Bergmann <arnd@arndb.de>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Len Brown <lenb@kernel.org>,
        Puranjay Mohan <puranjay12@gmail.com>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Jacopo De Simoi <wilderkde@gmail.com>
Date:   Mon, 27 Jul 2020 20:52:25 +0100
In-Reply-To: <20200727141438.GA1743062@bjorn-Precision-5520>
References: <20200727141438.GA1743062@bjorn-Precision-5520>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.36.4 (3.36.4-1.fc32) 
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, 2020-07-27 at 09:14 -0500, Bjorn Helgaas wrote:
> I don't know the connection between ASPM and package C-states, so I
> need to simplify this even more.  All I want to do right now is
> verify
> that if we don't have any outside influences on the ASPM
> configuration
> (eg, no manual changes and no udev rules), it stays the same across
> suspend/resume.

Basically this started from me observing deep package C-states weren't
being used, until I went and fiddled with the ASPM state of the
rtsx_pci card reader under sysfs -- so phenomenological poking on my
part.

> So let's read the ASPM state directly from the
> hardware like this:
> 
>   sudo lspci -vvs 00:1d.0 | egrep "^0|Lnk|L1|LTR|snoop"
>   sudo lspci -vvs 01:00   | egrep "^0|Lnk|L1|LTR|snoop"
> 
> Can you try that before and after suspend/resume?

I've attached these to the bugzilla entry at:

https://bugzilla.kernel.org/show_bug.cgi?id=208117

Spoiler: With no udev rules or suspend hooks, things are the same
before and after suspend/resume. One thing I do see (both before and
after) is that ASPM L0s and L1 is enabled for the card reader, but
disabled for the ethernet chip (does r8169 fiddle with ASPM too?).

[Oddly when I set ASPM (e.g. using udev) the lspci tools show ASPM
enabled after a suspend/resume, but still no deep package C-states
until I manually fiddle via sysfs on the card reader. Sorry if this
only muddies the water further!]

Thanks,
-James