Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 22 Jun 2023 18:06:07 -0500
From:   Bjorn Helgaas <helgaas@kernel.org>
To:     "Limonciello, Mario" <mario.limonciello@amd.com>
Cc:     Kai-Heng Feng <kai.heng.feng@canonical.com>, bhelgaas@google.com,
        Mika Westerberg <mika.westerberg@linux.intel.com>,
        Kuppuswamy Sathyanarayanan 
        <sathyanarayanan.kuppuswamy@linux.intel.com>,
        Vidya Sagar <vidyas@nvidia.com>,
        Michael Bottini <michael.a.bottini@linux.intel.com>,
        "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>,
        linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH] PCI/ASPM: Enable ASPM on external PCIe devices
Message-ID: <20230622230607.GA155247@bhelgaas>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <1b4b2c6c-8119-95fd-8958-dbbecc66510c@amd.com>
Precedence: bulk

On Tue, Jun 20, 2023 at 01:36:59PM -0500, Limonciello, Mario wrote:
> <snip>
> > > A variety of Intel chipsets don't support lane width switching
> > > or speed switching.? When ASPM has been enabled on a dGPU,
> > > these features are utilized and breakage ensues.
> > Maybe this helps explain all the completely unmaintainable ASPM
> > garbage in amdgpu and radeon.
> > 
> > If these devices are broken, we need quirks for them.
> 
> The problem is which device do you consider "broken"?
> The dGPU that uses these features when the platform advertises ASPM
> or the chipset which doesn't support the features that the device
> uses when ASPM is active?
> 
> With this problem I'm talking about the dGPU works fine on hosts
> that support these features.

Without more details about what's broken and when, I can't say.  What
I *think* is that a device that doesn't work per spec needs a quirk.
Typically it's a device that advertises a capability that doesn't work
correctly.

> > > > > I think the pragmatic way to approach it is to (essentially)
> > > > > apply the policy as BIOS defaults and allow overrides from
> > > > > that.
> > > >
> > > > Do you mean that when enumerating a device (at boot-time or
> > > > hot-add time), we would read the current ASPM config but not
> > > > change it?  And users could use the sysfs knobs to
> > > > enable/disable ASPM as desired?
> > >
> > > Yes.
> > >
> > Hot-added devices power up with ASPM disabled.  This policy would
> > mean the user has to explicitly enable it, which doesn't seem
> > practical to me.
>
> Could we maybe have the hot added devices follow the policy of
> the bridge they're connected to by default?
>
> > > > That wouldn't solve the problem Kai-Heng is trying to solve.
> > >
> > > Alone it wouldn't; but if you treated the i225 PCIe device
> > > connected to the system as a "quirk" to apply ASPM policy
> > > from the parent device to this child device it could.
> >
> > I want quirks for BROKEN devices.  Quirks for working hardware is a
> > maintenance nightmare.
>
> If you follow my idea of hot added devices the policy follows
> the parent would it work for the i225 PCIe device case?

That doesn't *sound* really robust to me because even if the default
config after hot-add works, the user can change things via sysfs, and
any configuration we set it to should work as well.  If there are
land-mines there, we need a quirk that prevents sysfs from running
into it.

Bjorn