Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 5 Sep 2022 10:50:33 +0300
From:   Mika Westerberg <mika.westerberg@linux.intel.com>
To:     Kai-Heng Feng <kai.heng.feng@canonical.com>
Cc:     andreas.noever@gmail.com, michael.jamet@intel.com,
        YehezkelShB@gmail.com, sanju.mehta@amd.com,
        mario.limonciello@amd.com, linux-usb@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH] thunderbolt: Resume PCIe bridges after switch is found
 on AMD USB4 controller
Message-ID: <YxWqSYDWe0NitSkL@black.fi.intel.com>
References: <20220905065622.1573811-1-kai.heng.feng@canonical.com>
 <YxWgGKIAvsxwSz85@black.fi.intel.com>
 <CAAd53p4iV=ne5bDGZ6FxE9bBUVoFh=eXF9_oMPvPzjVj=UVoog@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAAd53p4iV=ne5bDGZ6FxE9bBUVoFh=eXF9_oMPvPzjVj=UVoog@mail.gmail.com>
Precedence: bulk

On Mon, Sep 05, 2022 at 03:26:28PM +0800, Kai-Heng Feng wrote:
> Hi Mika,
> 
> On Mon, Sep 5, 2022 at 3:06 PM Mika Westerberg
> <mika.westerberg@linux.intel.com> wrote:
> >
> > Hi,
> >
> > On Mon, Sep 05, 2022 at 02:56:22PM +0800, Kai-Heng Feng wrote:
> > > AMD USB4 can not detect external PCIe devices like external NVMe when
> > > it's hotplugged, because card/link are not up:
> > >
> > > pcieport 0000:00:04.1: pciehp: pciehp_check_link_active: lnk_status = 1101
> >
> > I think the correct solution is then to block them from runtime
> > suspending entirely.
> 
> Do you mean disable runtime suspend completely? Or just block runtime
> suspend for a period?

Completely. The port should enter D3 if it cannot wake up and Linux does
not even enable runtime PM for such ports unless they declare
"HotPlugSupportInD3" in their ACPI description:

https://docs.microsoft.com/en-us/windows-hardware/drivers/pci/dsd-for-pcie-root-ports#identifying-pcie-root-ports-supporting-hot-plug-in-d3

So that property should not be there if they cannot wake up.

> > > Use `lspci` to resume pciehp bridges can find external devices.
> > >
> > > A long delay before checking card/link presence doesn't help, either.
> > > The only way to make the hotplug work is to enable pciehp interrupt and
> > > check card presence after the TB switch is added.
> > >
> > > Since the topology of USB4 and its PCIe bridges are siblings, hardcode
> > > the bridge ID so TBT driver can wake them up to check presence.
> >
> > Let's not add PCI things into TBT driver unless absolutely necessary.
> 
> OK. It's getting harder as different components are intertwined
> together on new hardwares...
> 
> >
> > At least on Intel hardware the PCIe hotplug is signaled by SCI when the
> > root port is in D3, I wonder if AMD has something similar.
> 
> Yes those root ports are resumed to D0 when something is plugged. They
> however fail to detect any externel PCIe devices.

Hmm, so you see the actual hotplug but the tunneled PCIe link may not be
detected? Does the PCIe "Card Present" (or Data Link Layer Active)
status change at all or is it always 0?