[+cc Alex and Mingchuang et al from
https://lore.kernel.org/r/[email protected]]
On Tue, Jan 19, 2021 at 04:14:10PM +0300, Mika Westerberg wrote:
> PCIe r5.0, sec 7.5.3.16 says that the downstream ports must reset the
> LTR enable bit if the link goes down (port goes DL_Down status). Now, if
> we had LTR previously enabled and the PCIe endpoint gets hot-removed and
> then hot-added back the ->ltr_path of the downstream port is still set
> but the port now does not have the LTR enable bit set anymore.
>
> For this reason check if the bridge upstream had LTR enabled previously
> and re-enable it before enabling LTR for the endpoint.
>
> Reported-by: Utkarsh H Patel <[email protected]>
> Signed-off-by: Mika Westerberg <[email protected]>
I think this and Mingchuang's patch, which is essentially identical,
are right and solves the problem for hot-remove/hot-add. In that
scenario we call pci_configure_ltr() on the hot-added device, and
with this patch, we'll re-enable LTR on the bridge leading to the new
device before enabling LTR on the new device itself.
But don't we have a similar problem if we simply do a Fundamental
Reset on a device? I think the reset path will restore the device's
state, including PCI_EXP_DEVCTL2, but it doesn't do anything with the
upstream bridge, does it?
So if a bridge and a device below it both have LTR enabled, can't we
have the following:
- bridge LTR enabled
- device LTR enabled
- reset device, e.g., via Secondary Bus Reset
- link goes down, bridge disables LTR
- link comes back up, LTR disabled in both bridge and device
- restore device state, including LTR enable
- device sends LTR message
- bridge reports Unsupported Request
> ---
> Previous version can be found here:
>
> https://lore.kernel.org/linux-pci/[email protected]/
>
> Changes from the previous version:
>
> * Corrected typos in the commit message
> * No need to call pcie_downstream_port()
>
> drivers/pci/probe.c | 17 ++++++++++++++++-
> 1 file changed, 16 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> index 0eb68b47354f..a4a8c0305fb9 100644
> --- a/drivers/pci/probe.c
> +++ b/drivers/pci/probe.c
> @@ -2153,7 +2153,7 @@ static void pci_configure_ltr(struct pci_dev *dev)
> {
> #ifdef CONFIG_PCIEASPM
> struct pci_host_bridge *host = pci_find_host_bridge(dev->bus);
> - struct pci_dev *bridge;
> + struct pci_dev *bridge = NULL;
> u32 cap, ctl;
>
> if (!pci_is_pcie(dev))
> @@ -2191,6 +2191,21 @@ static void pci_configure_ltr(struct pci_dev *dev)
> if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||
> ((bridge = pci_upstream_bridge(dev)) &&
> bridge->ltr_path)) {
> + /*
> + * Downstream ports reset the LTR enable bit when the
> + * link goes down (e.g on hot-remove) so re-enable the
> + * bit here if not set anymore.
> + * PCIe r5.0, sec 7.5.3.16.
> + */
> + if (bridge) {
> + pcie_capability_read_dword(bridge, PCI_EXP_DEVCTL2, &ctl);
> + if (!(ctl & PCI_EXP_DEVCTL2_LTR_EN)) {
> + pci_dbg(bridge, "re-enabling LTR\n");
> + pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
> + PCI_EXP_DEVCTL2_LTR_EN);
> + }
> + }
> + pci_dbg(dev, "enabling LTR\n");
> pcie_capability_set_word(dev, PCI_EXP_DEVCTL2,
> PCI_EXP_DEVCTL2_LTR_EN);
> dev->ltr_path = 1;
> --
> 2.29.2
>
On Thu, 2021-01-21 at 16:31 -0600, Bjorn Helgaas wrote:
> [+cc Alex and Mingchuang et al from
> https://lore.kernel.org/r/[email protected]]
>
> On Tue, Jan 19, 2021 at 04:14:10PM +0300, Mika Westerberg wrote:
> > PCIe r5.0, sec 7.5.3.16 says that the downstream ports must reset the
> > LTR enable bit if the link goes down (port goes DL_Down status). Now, if
> > we had LTR previously enabled and the PCIe endpoint gets hot-removed and
> > then hot-added back the ->ltr_path of the downstream port is still set
> > but the port now does not have the LTR enable bit set anymore.
> >
> > For this reason check if the bridge upstream had LTR enabled previously
> > and re-enable it before enabling LTR for the endpoint.
> >
> > Reported-by: Utkarsh H Patel <[email protected]>
> > Signed-off-by: Mika Westerberg <[email protected]>
>
> I think this and Mingchuang's patch, which is essentially identical,
> are right and solves the problem for hot-remove/hot-add. In that
> scenario we call pci_configure_ltr() on the hot-added device, and
> with this patch, we'll re-enable LTR on the bridge leading to the new
> device before enabling LTR on the new device itself.
>
> But don't we have a similar problem if we simply do a Fundamental
> Reset on a device? I think the reset path will restore the device's
> state, including PCI_EXP_DEVCTL2, but it doesn't do anything with the
> upstream bridge, does it?
>
Yes. I think the same problem exists under such scenario, and that’s the
issue my patch intends to resolve.
I also prepared a v2 patch for review(update the patch description).
Shall I submit the v2 patch for review?
> So if a bridge and a device below it both have LTR enabled, can't we
> have the following:
>
> - bridge LTR enabled
> - device LTR enabled
> - reset device, e.g., via Secondary Bus Reset
> - link goes down, bridge disables LTR
> - link comes back up, LTR disabled in both bridge and device
> - restore device state, including LTR enable
> - device sends LTR message
> - bridge reports Unsupported Request
>
> > ---
> > Previous version can be found here:
> >
> > https://lore.kernel.org/linux-pci/[email protected]/
> >
> > Changes from the previous version:
> >
> > * Corrected typos in the commit message
> > * No need to call pcie_downstream_port()
> >
> > drivers/pci/probe.c | 17 ++++++++++++++++-
> > 1 file changed, 16 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> > index 0eb68b47354f..a4a8c0305fb9 100644
> > --- a/drivers/pci/probe.c
> > +++ b/drivers/pci/probe.c
> > @@ -2153,7 +2153,7 @@ static void pci_configure_ltr(struct pci_dev *dev)
> > {
> > #ifdef CONFIG_PCIEASPM
> > struct pci_host_bridge *host = pci_find_host_bridge(dev->bus);
> > - struct pci_dev *bridge;
> > + struct pci_dev *bridge = NULL;
> > u32 cap, ctl;
> >
> > if (!pci_is_pcie(dev))
> > @@ -2191,6 +2191,21 @@ static void pci_configure_ltr(struct pci_dev *dev)
> > if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||
> > ((bridge = pci_upstream_bridge(dev)) &&
> > bridge->ltr_path)) {
> > + /*
> > + * Downstream ports reset the LTR enable bit when the
> > + * link goes down (e.g on hot-remove) so re-enable the
> > + * bit here if not set anymore.
> > + * PCIe r5.0, sec 7.5.3.16.
> > + */
> > + if (bridge) {
> > + pcie_capability_read_dword(bridge, PCI_EXP_DEVCTL2, &ctl);
> > + if (!(ctl & PCI_EXP_DEVCTL2_LTR_EN)) {
> > + pci_dbg(bridge, "re-enabling LTR\n");
> > + pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
> > + PCI_EXP_DEVCTL2_LTR_EN);
> > + }
> > + }
> > + pci_dbg(dev, "enabling LTR\n");
> > pcie_capability_set_word(dev, PCI_EXP_DEVCTL2,
> > PCI_EXP_DEVCTL2_LTR_EN);
> > dev->ltr_path = 1;
> > --
> > 2.29.2
> >
Hi,
On Fri, Jan 22, 2021 at 03:03:11PM +0800, Mingchuang Qiao wrote:
> On Thu, 2021-01-21 at 16:31 -0600, Bjorn Helgaas wrote:
> > [+cc Alex and Mingchuang et al from
> > https://lore.kernel.org/r/[email protected]]
> >
> > On Tue, Jan 19, 2021 at 04:14:10PM +0300, Mika Westerberg wrote:
> > > PCIe r5.0, sec 7.5.3.16 says that the downstream ports must reset the
> > > LTR enable bit if the link goes down (port goes DL_Down status). Now, if
> > > we had LTR previously enabled and the PCIe endpoint gets hot-removed and
> > > then hot-added back the ->ltr_path of the downstream port is still set
> > > but the port now does not have the LTR enable bit set anymore.
> > >
> > > For this reason check if the bridge upstream had LTR enabled previously
> > > and re-enable it before enabling LTR for the endpoint.
> > >
> > > Reported-by: Utkarsh H Patel <[email protected]>
> > > Signed-off-by: Mika Westerberg <[email protected]>
> >
> > I think this and Mingchuang's patch, which is essentially identical,
> > are right and solves the problem for hot-remove/hot-add. In that
> > scenario we call pci_configure_ltr() on the hot-added device, and
> > with this patch, we'll re-enable LTR on the bridge leading to the new
> > device before enabling LTR on the new device itself.
> >
> > But don't we have a similar problem if we simply do a Fundamental
> > Reset on a device? I think the reset path will restore the device's
> > state, including PCI_EXP_DEVCTL2, but it doesn't do anything with the
> > upstream bridge, does it?
> >
>
> Yes. I think the same problem exists under such scenario, and that’s the
> issue my patch intends to resolve.
> I also prepared a v2 patch for review(update the patch description).
> Shall I submit the v2 patch for review?
I looked at your patch and indeed it is essentially doing the same as
this one. So let's forget this patch and go forward with yours :)
Would you like to expand your patch to handle the reset case too that
Bjorn desribes below?
> > So if a bridge and a device below it both have LTR enabled, can't we
> > have the following:
> >
> > - bridge LTR enabled
> > - device LTR enabled
> > - reset device, e.g., via Secondary Bus Reset
> > - link goes down, bridge disables LTR
> > - link comes back up, LTR disabled in both bridge and device
> > - restore device state, including LTR enable
> > - device sends LTR message
> > - bridge reports Unsupported Request
On Fri, Jan 22, 2021 at 03:03:11PM +0800, Mingchuang Qiao wrote:
> On Thu, 2021-01-21 at 16:31 -0600, Bjorn Helgaas wrote:
> > [+cc Alex and Mingchuang et al from
> > https://lore.kernel.org/r/[email protected]]
> >
> > On Tue, Jan 19, 2021 at 04:14:10PM +0300, Mika Westerberg wrote:
> > > PCIe r5.0, sec 7.5.3.16 says that the downstream ports must reset the
> > > LTR enable bit if the link goes down (port goes DL_Down status). Now, if
> > > we had LTR previously enabled and the PCIe endpoint gets hot-removed and
> > > then hot-added back the ->ltr_path of the downstream port is still set
> > > but the port now does not have the LTR enable bit set anymore.
> > >
> > > For this reason check if the bridge upstream had LTR enabled previously
> > > and re-enable it before enabling LTR for the endpoint.
> > >
> > > Reported-by: Utkarsh H Patel <[email protected]>
> > > Signed-off-by: Mika Westerberg <[email protected]>
> >
> > I think this and Mingchuang's patch, which is essentially identical,
> > are right and solves the problem for hot-remove/hot-add. In that
> > scenario we call pci_configure_ltr() on the hot-added device, and
> > with this patch, we'll re-enable LTR on the bridge leading to the new
> > device before enabling LTR on the new device itself.
> >
> > But don't we have a similar problem if we simply do a Fundamental
> > Reset on a device? I think the reset path will restore the device's
> > state, including PCI_EXP_DEVCTL2, but it doesn't do anything with the
> > upstream bridge, does it?
>
> Yes. I think the same problem exists under such scenario, and that’s the
> issue my patch intends to resolve.
> I also prepared a v2 patch for review(update the patch description).
> Shall I submit the v2 patch for review?
How does your patch solve this for the reset path? I don't think we
call pci_configure_ltr() when we reset a device.
> > So if a bridge and a device below it both have LTR enabled, can't we
> > have the following:
> >
> > - bridge LTR enabled
> > - device LTR enabled
> > - reset device, e.g., via Secondary Bus Reset
> > - link goes down, bridge disables LTR
> > - link comes back up, LTR disabled in both bridge and device
> > - restore device state, including LTR enable
> > - device sends LTR message
> > - bridge reports Unsupported Request
> >
> > > ---
> > > Previous version can be found here:
> > >
> > > https://lore.kernel.org/linux-pci/[email protected]/
> > >
> > > Changes from the previous version:
> > >
> > > * Corrected typos in the commit message
> > > * No need to call pcie_downstream_port()
> > >
> > > drivers/pci/probe.c | 17 ++++++++++++++++-
> > > 1 file changed, 16 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> > > index 0eb68b47354f..a4a8c0305fb9 100644
> > > --- a/drivers/pci/probe.c
> > > +++ b/drivers/pci/probe.c
> > > @@ -2153,7 +2153,7 @@ static void pci_configure_ltr(struct pci_dev *dev)
> > > {
> > > #ifdef CONFIG_PCIEASPM
> > > struct pci_host_bridge *host = pci_find_host_bridge(dev->bus);
> > > - struct pci_dev *bridge;
> > > + struct pci_dev *bridge = NULL;
> > > u32 cap, ctl;
> > >
> > > if (!pci_is_pcie(dev))
> > > @@ -2191,6 +2191,21 @@ static void pci_configure_ltr(struct pci_dev *dev)
> > > if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||
> > > ((bridge = pci_upstream_bridge(dev)) &&
> > > bridge->ltr_path)) {
> > > + /*
> > > + * Downstream ports reset the LTR enable bit when the
> > > + * link goes down (e.g on hot-remove) so re-enable the
> > > + * bit here if not set anymore.
> > > + * PCIe r5.0, sec 7.5.3.16.
> > > + */
> > > + if (bridge) {
> > > + pcie_capability_read_dword(bridge, PCI_EXP_DEVCTL2, &ctl);
> > > + if (!(ctl & PCI_EXP_DEVCTL2_LTR_EN)) {
> > > + pci_dbg(bridge, "re-enabling LTR\n");
> > > + pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
> > > + PCI_EXP_DEVCTL2_LTR_EN);
> > > + }
> > > + }
> > > + pci_dbg(dev, "enabling LTR\n");
> > > pcie_capability_set_word(dev, PCI_EXP_DEVCTL2,
> > > PCI_EXP_DEVCTL2_LTR_EN);
> > > dev->ltr_path = 1;
> > > --
> > > 2.29.2
> > >
>
On Fri, 2021-01-22 at 07:20 -0600, Bjorn Helgaas wrote:
> On Fri, Jan 22, 2021 at 03:03:11PM +0800, Mingchuang Qiao wrote:
> > On Thu, 2021-01-21 at 16:31 -0600, Bjorn Helgaas wrote:
> > > [+cc Alex and Mingchuang et al from
> > > https://lore.kernel.org/r/[email protected]]
> > >
> > > On Tue, Jan 19, 2021 at 04:14:10PM +0300, Mika Westerberg wrote:
> > > > PCIe r5.0, sec 7.5.3.16 says that the downstream ports must reset the
> > > > LTR enable bit if the link goes down (port goes DL_Down status). Now, if
> > > > we had LTR previously enabled and the PCIe endpoint gets hot-removed and
> > > > then hot-added back the ->ltr_path of the downstream port is still set
> > > > but the port now does not have the LTR enable bit set anymore.
> > > >
> > > > For this reason check if the bridge upstream had LTR enabled previously
> > > > and re-enable it before enabling LTR for the endpoint.
> > > >
> > > > Reported-by: Utkarsh H Patel <[email protected]>
> > > > Signed-off-by: Mika Westerberg <[email protected]>
> > >
> > > I think this and Mingchuang's patch, which is essentially identical,
> > > are right and solves the problem for hot-remove/hot-add. In that
> > > scenario we call pci_configure_ltr() on the hot-added device, and
> > > with this patch, we'll re-enable LTR on the bridge leading to the new
> > > device before enabling LTR on the new device itself.
> > >
> > > But don't we have a similar problem if we simply do a Fundamental
> > > Reset on a device? I think the reset path will restore the device's
> > > state, including PCI_EXP_DEVCTL2, but it doesn't do anything with the
> > > upstream bridge, does it?
> >
> > Yes. I think the same problem exists under such scenario, and that’s the
> > issue my patch intends to resolve.
> > I also prepared a v2 patch for review(update the patch description).
> > Shall I submit the v2 patch for review?
>
> How does your patch solve this for the reset path? I don't think we
> call pci_configure_ltr() when we reset a device.
>
Sorry, I misunderstand the reset path. When we do a Fundamental Reset on
a device, we can trigger device removal and rescan flow to restore the
device. In device rescan flow, pci_configure_ltr() will be invoked. I
regard the "remove and rescan flow" as a part of reset path for this
case :)
If we restore device just with pci_restore_state() in device driver
after device resets, the LTR problem also exists due to
pci_restore_state() does nothing with upstream bridge. In next patch, I
would like to re-enable LTR for upstream bridge before restoring
device's PCI_EXP_DEVCTL2 if it is needed.
> > > So if a bridge and a device below it both have LTR enabled, can't we
> > > have the following:
> > >
> > > - bridge LTR enabled
> > > - device LTR enabled
> > > - reset device, e.g., via Secondary Bus Reset
> > > - link goes down, bridge disables LTR
> > > - link comes back up, LTR disabled in both bridge and device
> > > - restore device state, including LTR enable
> > > - device sends LTR message
> > > - bridge reports Unsupported Request
> > >
> > > > ---
> > > > Previous version can be found here:
> > > >
> > > > https://lore.kernel.org/linux-pci/[email protected]/
> > > >
> > > > Changes from the previous version:
> > > >
> > > > * Corrected typos in the commit message
> > > > * No need to call pcie_downstream_port()
> > > >
> > > > drivers/pci/probe.c | 17 ++++++++++++++++-
> > > > 1 file changed, 16 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> > > > index 0eb68b47354f..a4a8c0305fb9 100644
> > > > --- a/drivers/pci/probe.c
> > > > +++ b/drivers/pci/probe.c
> > > > @@ -2153,7 +2153,7 @@ static void pci_configure_ltr(struct pci_dev *dev)
> > > > {
> > > > #ifdef CONFIG_PCIEASPM
> > > > struct pci_host_bridge *host = pci_find_host_bridge(dev->bus);
> > > > - struct pci_dev *bridge;
> > > > + struct pci_dev *bridge = NULL;
> > > > u32 cap, ctl;
> > > >
> > > > if (!pci_is_pcie(dev))
> > > > @@ -2191,6 +2191,21 @@ static void pci_configure_ltr(struct pci_dev *dev)
> > > > if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||
> > > > ((bridge = pci_upstream_bridge(dev)) &&
> > > > bridge->ltr_path)) {
> > > > + /*
> > > > + * Downstream ports reset the LTR enable bit when the
> > > > + * link goes down (e.g on hot-remove) so re-enable the
> > > > + * bit here if not set anymore.
> > > > + * PCIe r5.0, sec 7.5.3.16.
> > > > + */
> > > > + if (bridge) {
> > > > + pcie_capability_read_dword(bridge, PCI_EXP_DEVCTL2, &ctl);
> > > > + if (!(ctl & PCI_EXP_DEVCTL2_LTR_EN)) {
> > > > + pci_dbg(bridge, "re-enabling LTR\n");
> > > > + pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
> > > > + PCI_EXP_DEVCTL2_LTR_EN);
> > > > + }
> > > > + }
> > > > + pci_dbg(dev, "enabling LTR\n");
> > > > pcie_capability_set_word(dev, PCI_EXP_DEVCTL2,
> > > > PCI_EXP_DEVCTL2_LTR_EN);
> > > > dev->ltr_path = 1;
> > > > --
> > > > 2.29.2
> > > >
> >