Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220905065622.1573811-1-kai.heng.feng@canonical.com>
 <YxWgGKIAvsxwSz85@black.fi.intel.com> <CAAd53p4iV=ne5bDGZ6FxE9bBUVoFh=eXF9_oMPvPzjVj=UVoog@mail.gmail.com>
 <YxWqSYDWe0NitSkL@black.fi.intel.com>
In-Reply-To: <YxWqSYDWe0NitSkL@black.fi.intel.com>
From:   Kai-Heng Feng <kai.heng.feng@canonical.com>
Date:   Mon, 5 Sep 2022 23:21:36 +0800
Message-ID: <CAAd53p6bSmTPavjA0v6tybc6=HrwiDn0JGzXwVOG_m5EVw1p1w@mail.gmail.com>
Subject: Re: [PATCH] thunderbolt: Resume PCIe bridges after switch is found on
 AMD USB4 controller
To:     Mika Westerberg <mika.westerberg@linux.intel.com>
Cc:     andreas.noever@gmail.com, michael.jamet@intel.com,
        YehezkelShB@gmail.com, sanju.mehta@amd.com,
        mario.limonciello@amd.com, linux-usb@vger.kernel.org,
        linux-kernel@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Mon, Sep 5, 2022 at 3:50 PM Mika Westerberg
<mika.westerberg@linux.intel.com> wrote:
>
> On Mon, Sep 05, 2022 at 03:26:28PM +0800, Kai-Heng Feng wrote:
> > Hi Mika,
> >
> > On Mon, Sep 5, 2022 at 3:06 PM Mika Westerberg
> > <mika.westerberg@linux.intel.com> wrote:
> > >
> > > Hi,
> > >
> > > On Mon, Sep 05, 2022 at 02:56:22PM +0800, Kai-Heng Feng wrote:
> > > > AMD USB4 can not detect external PCIe devices like external NVMe when
> > > > it's hotplugged, because card/link are not up:
> > > >
> > > > pcieport 0000:00:04.1: pciehp: pciehp_check_link_active: lnk_status = 1101
> > >
> > > I think the correct solution is then to block them from runtime
> > > suspending entirely.
> >
> > Do you mean disable runtime suspend completely? Or just block runtime
> > suspend for a period?
>
> Completely. The port should enter D3 if it cannot wake up and Linux does
> not even enable runtime PM for such ports unless they declare
> "HotPlugSupportInD3" in their ACPI description:
>
> https://docs.microsoft.com/en-us/windows-hardware/drivers/pci/dsd-for-pcie-root-ports#identifying-pcie-root-ports-supporting-hot-plug-in-d3
>
> So that property should not be there if they cannot wake up.

The port can wake up but card/link isn't up yet.

>
> > > > Use `lspci` to resume pciehp bridges can find external devices.
> > > >
> > > > A long delay before checking card/link presence doesn't help, either.
> > > > The only way to make the hotplug work is to enable pciehp interrupt and
> > > > check card presence after the TB switch is added.
> > > >
> > > > Since the topology of USB4 and its PCIe bridges are siblings, hardcode
> > > > the bridge ID so TBT driver can wake them up to check presence.
> > >
> > > Let's not add PCI things into TBT driver unless absolutely necessary.
> >
> > OK. It's getting harder as different components are intertwined
> > together on new hardwares...
> >
> > >
> > > At least on Intel hardware the PCIe hotplug is signaled by SCI when the
> > > root port is in D3, I wonder if AMD has something similar.
> >
> > Yes those root ports are resumed to D0 when something is plugged. They
> > however fail to detect any externel PCIe devices.
>
> Hmm, so you see the actual hotplug but the tunneled PCIe link may not be
> detected? Does the PCIe "Card Present" (or Data Link Layer Active)
> status change at all or is it always 0?

It changes only after tb_switch_add() is called.

Kai-Heng