Received-SPF: pass (google.com: domain of linux-kernel+bounces-129845-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Wed, 3 Apr 2024 19:02:17 +0530
From: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
To: Niklas Cassel <cassel@kernel.org>
Cc: Lorenzo Pieralisi <lpieralisi@kernel.org>,
	Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= <kw@linux.com>,
	Rob Herring <robh@kernel.org>, Bjorn Helgaas <bhelgaas@google.com>,
	Kishon Vijay Abraham I <kishon@kernel.org>,
	Thierry Reding <thierry.reding@gmail.com>,
	Jonathan Hunter <jonathanh@nvidia.com>,
	Jingoo Han <jingoohan1@gmail.com>, linux-pci@vger.kernel.org,
	linux-arm-msm@vger.kernel.org, linux-kernel@vger.kernel.org,
	mhi@lists.linux.dev, linux-tegra@vger.kernel.org
Subject: Re: [PATCH v2 10/10] PCI: qcom: Implement shutdown() callback to
 properly reset the endpoint devices
Message-ID: <20240403133217.GK25309@thinkpad>
References: <20240401-pci-epf-rework-v2-0-970dbe90b99d@linaro.org>
 <20240401-pci-epf-rework-v2-10-970dbe90b99d@linaro.org>
 <ZgvpnqdjQ39JMRiV@ryzen>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <ZgvpnqdjQ39JMRiV@ryzen>

On Tue, Apr 02, 2024 at 01:18:54PM +0200, Niklas Cassel wrote:
> On Mon, Apr 01, 2024 at 09:20:36PM +0530, Manivannan Sadhasivam wrote:
> > PCIe host controller drivers are supposed to properly reset the endpoint
> > devices during host shutdown/reboot. Currently, Qcom driver doesn't do
> > anything during host shutdown/reboot, resulting in both PERST# and refclk
> > getting disabled at the same time. This prevents the endpoint device
> > firmware to properly reset the state machine. Because, if the refclk is
> > cutoff immediately along with PERST#, access to device specific registers
> > within the endpoint will result in a firmware crash.
> > 
> > To address this issue, let's call qcom_pcie_host_deinit() inside the
> > shutdown callback, that asserts PERST# and then cuts off the refclk with a
> > delay of 1ms, thus allowing the endpoint device firmware to properly
> > cleanup the state machine.
> 
> Hm... a QCOM EP device could be attached to any of the PCIe RC drivers that
> we have in the kernel, so it seems a bit weird to fix this problem by
> patching the QCOM RC driver only.
> 
> Which DBI call is it that causes this problem during perst assert on EP side?
> 
> I assume that it is pci-epf-test:deinit() callback that calls
> pci_epc_clear_bar(), which calls dw_pcie_ep_clear_bar(), which will both:
> -clear local data structures, e.g.
> ep->epf_bar[bar] = NULL;
> ep->bar_to_atu[bar] = 0;
> 
> but also call:
> __dw_pcie_ep_reset_bar()
> dw_pcie_disable_atu()
> 
> 
> Do we perhaps need to redesign the .deinit EPF callback?
> 
> Considering that we know that .deinit() will only be called on platforms
> where there will be a fundamental core reset, I guess we could do something
> like introduce a __dw_pcie_ep_clear_bar() which will only clear the local
> data structures. (It might not need to do any DBI writes, since the
> fundamental core reset should have reset all values.)
> 
> Or perhaps instead of letting pci_epf_test_epc_deinit() call
> pci_epf_test_clear_bar()/__pci_epf_test_clear_bar() directly, perhaps let
> pci_epf_test_epc_deinit() call add a .deinit()/.cleanup() defined in the
> EPC driver.
> 
> This EPC .deinit()/.cleanup() callback would then only clear the
> local data structures (no DBI writes...).
> 
> Something like that?
> 

It is not just about the EPF test driver. A function driver may need to do many
things to properly reset the state machine. Like in the case of MHI driver, it
needs to reset channel state, mask interrupts etc... and all requires writing to
some registers. So certainly there should be some time before cutting off the
refclk.

On x86 host machines, I can see that there is enough time before the host cuts
off the refclk.

But it is unfortunate that the PCIe spec didn't define it though.

- Mani

> 
> > 
> > Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> > ---
> >  drivers/pci/controller/dwc/pcie-qcom.c | 8 ++++++++
> >  1 file changed, 8 insertions(+)
> > 
> > diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c
> > index 14772edcf0d3..b2803978c0ad 100644
> > --- a/drivers/pci/controller/dwc/pcie-qcom.c
> > +++ b/drivers/pci/controller/dwc/pcie-qcom.c
> > @@ -1655,6 +1655,13 @@ static int qcom_pcie_resume_noirq(struct device *dev)
> >  	return 0;
> >  }
> >  
> > +static void qcom_pcie_shutdown(struct platform_device *pdev)
> > +{
> > +	struct qcom_pcie *pcie = platform_get_drvdata(pdev);
> > +
> > +	qcom_pcie_host_deinit(&pcie->pci->pp);
> > +}
> > +
> >  static const struct of_device_id qcom_pcie_match[] = {
> >  	{ .compatible = "qcom,pcie-apq8064", .data = &cfg_2_1_0 },
> >  	{ .compatible = "qcom,pcie-apq8084", .data = &cfg_1_0_0 },
> > @@ -1708,5 +1715,6 @@ static struct platform_driver qcom_pcie_driver = {
> >  		.pm = &qcom_pcie_pm_ops,
> >  		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
> >  	},
> > +	.shutdown = qcom_pcie_shutdown,
> >  };
> >  builtin_platform_driver(qcom_pcie_driver);
> > 
> > -- 
> > 2.25.1
> > 

-- 
மணிவண்ணன் சதாசிவம்