Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38;
Message-ID: <f2d1476199e974dd1d303b22c25594f066e85de5.camel@linux.intel.com>
Subject: Re: [PATCH 1/6] PCI/ASPM: Add locked helper for enabling link state
From:   "David E. Box" <david.e.box@linux.intel.com>
Reply-To: david.e.box@linux.intel.com
To:     Johan Hovold <johan+linaro@kernel.org>,
        Lorenzo Pieralisi <lpieralisi@kernel.org>,
        Krzysztof =?UTF-8?Q?Wilczy=C5=84ski?= <kw@linux.com>,
        Bjorn Helgaas <bhelgaas@google.com>
Cc:     Andy Gross <agross@kernel.org>,
        Bjorn Andersson <andersson@kernel.org>,
        Konrad Dybcio <konrad.dybcio@linaro.org>,
        Manivannan Sadhasivam <mani@kernel.org>,
        Rob Herring <robh@kernel.org>,
        Nirmal Patel <nirmal.patel@linux.intel.com>,
        Jonathan Derrick <jonathan.derrick@linux.dev>,
        linux-arm-msm@vger.kernel.org, linux-pci@vger.kernel.org,
        linux-kernel@vger.kernel.org, stable@vger.kernel.org
Date:   Fri, 17 Nov 2023 16:00:46 -0800
In-Reply-To: <20231114135553.32301-2-johan+linaro@kernel.org>
References: <20231114135553.32301-1-johan+linaro@kernel.org>
         <20231114135553.32301-2-johan+linaro@kernel.org>
Organization: David E. Box
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.48.4 (3.48.4-1.fc38) 
MIME-Version: 1.0
Precedence: bulk

On Tue, 2023-11-14 at 14:55 +0100, Johan Hovold wrote:
> Add a helper for enabling link states that can be used in contexts where
> a pci_bus_sem read lock is already held (e.g. from pci_walk_bus()).
>=20
> This helper will be used to fix a couple of potential deadlocks where
> the current helper is called with the lock already held, hence the CC
> stable tag.

Thanks for sending a patch to address this. It had fallen off my radar.

>=20
> Fixes: f492edb40b54 ("PCI: vmd: Add quirk to configure PCIe ASPM and LTR"=
)
> Cc: stable@vger.kernel.org=C2=A0=C2=A0=C2=A0# 6.3
> Cc: Michael Bottini <michael.a.bottini@linux.intel.com>
> Cc: David E. Box <david.e.box@linux.intel.com>
> Signed-off-by: Johan Hovold <johan+linaro@kernel.org>
> ---
> =C2=A0drivers/pci/pcie/aspm.c | 53 +++++++++++++++++++++++++++++++-------=
---
> =C2=A0include/linux/pci.h=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0 3 +++
> =C2=A02 files changed, 43 insertions(+), 13 deletions(-)
>=20
> diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
> index 50b04ae5c394..8cf8cc2d6bba 100644
> --- a/drivers/pci/pcie/aspm.c
> +++ b/drivers/pci/pcie/aspm.c
> @@ -1109,17 +1109,7 @@ int pci_disable_link_state(struct pci_dev *pdev, i=
nt
> state)
> =C2=A0}
> =C2=A0EXPORT_SYMBOL(pci_disable_link_state);
> =C2=A0
> -/**
> - * pci_enable_link_state - Clear and set the default device link state s=
o
> that
> - * the link may be allowed to enter the specified states. Note that if t=
he
> - * BIOS didn't grant ASPM control to the OS, this does nothing because w=
e
> can't
> - * touch the LNKCTL register. Also note that this does not enable states
> - * disabled by pci_disable_link_state(). Return 0 or a negative errno.
> - *
> - * @pdev: PCI device
> - * @state: Mask of ASPM link states to enable
> - */
> -int pci_enable_link_state(struct pci_dev *pdev, int state)
> +static int __pci_enable_link_state(struct pci_dev *pdev, int state, bool
> locked)
> =C2=A0{
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0struct pcie_link_state *l=
ink =3D pcie_aspm_get_link(pdev);
> =C2=A0
> @@ -1136,7 +1126,8 @@ int pci_enable_link_state(struct pci_dev *pdev, int
> state)
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0return -EPERM;
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0}
> =C2=A0
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0down_read(&pci_bus_sem);
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if (!locked)
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0down_read(&pci_bus_sem);

This solution is similar to the original one I proposed [1]. It just create=
s a
separate locked function rather than using a flag. While the API is consist=
ent
with pci_disable_link_state_locked(), its usage is not. The vmd driver call=
s
pci_enable_link_state() from pci_walk_bus() which was problematic for Bjorn=
 [2].
I owed an attempt to implement this as a fixup. However, now
qcom_pcie_enabled_aspm() is also using it from pci_walk_bus(). The fixup co=
uld
make sense for VMD since there it's fixing missing BIOS settings, but I'm n=
ot
sure about qcom. Bjorn, do you still see issues with the use in pci_bus_wal=
k()?

[1]
https://lore.kernel.org/lkml/20230321233849.3408339-1-david.e.box@linux.int=
el.com/

[2] https://lore.kernel.org/lkml/20230322205702.GA2493123@bhelgaas/

David

> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0mutex_lock(&aspm_lock);
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0link->aspm_default =3D 0;
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if (state & PCIE_LINK_STA=
TE_L0S)
> @@ -1157,12 +1148,48 @@ int pci_enable_link_state(struct pci_dev *pdev, i=
nt
> state)
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0link->clkpm_default =3D (=
state & PCIE_LINK_STATE_CLKPM) ? 1 : 0;
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0pcie_set_clkpm(link, poli=
cy_to_clkpm_state(link));
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0mutex_unlock(&aspm_lock);
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0up_read(&pci_bus_sem);
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if (!locked)
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0up_read(&pci_bus_sem);
> =C2=A0
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return 0;
> =C2=A0}
> +
> +/**
> + * pci_enable_link_state - Clear and set the default device link state s=
o
> that
> + * the link may be allowed to enter the specified states. Note that if t=
he
> + * BIOS didn't grant ASPM control to the OS, this does nothing because w=
e
> can't
> + * touch the LNKCTL register. Also note that this does not enable states
> + * disabled by pci_disable_link_state(). Return 0 or a negative errno.
> + *
> + * @pdev: PCI device
> + * @state: Mask of ASPM link states to enable
> + */
> +int pci_enable_link_state(struct pci_dev *pdev, int state)
> +{
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return __pci_enable_link_state=
(pdev, state, false);
> +}
> =C2=A0EXPORT_SYMBOL(pci_enable_link_state);
> =C2=A0
> +/**
> + * pci_enable_link_state - Clear and set the default device link state s=
o
> that
> + * the link may be allowed to enter the specified states. Note that if t=
he
> + * BIOS didn't grant ASPM control to the OS, this does nothing because w=
e
> can't
> + * touch the LNKCTL register. Also note that this does not enable states
> + * disabled by pci_disable_link_state(). Return 0 or a negative errno.
> + *
> + * @pdev: PCI device
> + * @state: Mask of ASPM link states to enable
> + *
> + * Context: Caller holds pci_bus_sem read lock.
> + */
> +int pci_enable_link_state_locked(struct pci_dev *pdev, int state)
> +{
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lockdep_assert_held_read(&pci_=
bus_sem);
> +
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return __pci_enable_link_state=
(pdev, state, true);
> +}
> +EXPORT_SYMBOL(pci_enable_link_state_locked);
> +
> =C2=A0static int pcie_aspm_set_policy(const char *val,
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0const struct kernel_param *kp)
> =C2=A0{
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index 60ca768bc867..dea043bc1e38 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -1829,6 +1829,7 @@ extern bool pcie_ports_native;
> =C2=A0int pci_disable_link_state(struct pci_dev *pdev, int state);
> =C2=A0int pci_disable_link_state_locked(struct pci_dev *pdev, int state);
> =C2=A0int pci_enable_link_state(struct pci_dev *pdev, int state);
> +int pci_enable_link_state_locked(struct pci_dev *pdev, int state);
> =C2=A0void pcie_no_aspm(void);
> =C2=A0bool pcie_aspm_support_enabled(void);
> =C2=A0bool pcie_aspm_enabled(struct pci_dev *pdev);
> @@ -1839,6 +1840,8 @@ static inline int pci_disable_link_state_locked(str=
uct
> pci_dev *pdev, int state)
> =C2=A0{ return 0; }
> =C2=A0static inline int pci_enable_link_state(struct pci_dev *pdev, int s=
tate)
> =C2=A0{ return 0; }
> +static inline int pci_enable_link_state_locked(struct pci_dev *pdev, int
> state)
> +{ return 0; }
> =C2=A0static inline void pcie_no_aspm(void) { }
> =C2=A0static inline bool pcie_aspm_support_enabled(void) { return false; =
}
> =C2=A0static inline bool pcie_aspm_enabled(struct pci_dev *pdev) { return=
 false; }