Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 8 May 2023 16:14:30 -0500
From:   Bjorn Helgaas <helgaas@kernel.org>
To:     Siddharth Vadapalli <s-vadapalli@ti.com>
Cc:     tjoseph@cadence.com, lpieralisi@kernel.org, robh@kernel.org,
        kw@linux.com, bhelgaas@google.com, nadeem@cadence.com,
        linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org, vigneshr@ti.com, srk@ti.com,
        nm@ti.com
Subject: Re: [PATCH v2] PCI: cadence: Fix Gen2 Link Retraining process
Message-ID: <20230508211430.GA1185556@bhelgaas>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230315070800.1615527-1-s-vadapalli@ti.com>
Precedence: bulk

On Wed, Mar 15, 2023 at 12:38:00PM +0530, Siddharth Vadapalli wrote:
> The Link Retraining process is initiated to account for the Gen2 defect in
> the Cadence PCIe controller in J721E SoC. The errata corresponding to this
> is i2085, documented at:
> https://www.ti.com/lit/er/sprz455c/sprz455c.pdf
> 
> The existing workaround implemented for the errata waits for the Data Link
> initialization to complete and assumes that the link retraining process
> at the Physical Layer has completed. However, it is possible that the
> Physical Layer training might be ongoing as indicated by the
> PCI_EXP_LNKSTA_LT bit in the PCI_EXP_LNKSTA register.
> 
> Fix the existing workaround, to ensure that the Physical Layer training
> has also completed, in addition to the Data Link initialization.
> 
> Fixes: 4740b969aaf5 ("PCI: cadence: Retrain Link to work around Gen2 training defect")
> Signed-off-by: Siddharth Vadapalli <s-vadapalli@ti.com>
> Reviewed-by: Vignesh Raghavendra <vigneshr@ti.com>
> ---
> Changes from v1:
> 1. Collect Reviewed-by tag from Vignesh Raghavendra.
> 2. Rebase on next-20230315.
> 
> v1:
> https://lore.kernel.org/r/20230102075656.260333-1-s-vadapalli@ti.com
> 
>  .../controller/cadence/pcie-cadence-host.c    | 27 +++++++++++++++++++
>  1 file changed, 27 insertions(+)
> 
> diff --git a/drivers/pci/controller/cadence/pcie-cadence-host.c b/drivers/pci/controller/cadence/pcie-cadence-host.c
> index 940c7dd701d6..5b14f7ee3c79 100644
> --- a/drivers/pci/controller/cadence/pcie-cadence-host.c
> +++ b/drivers/pci/controller/cadence/pcie-cadence-host.c
> @@ -12,6 +12,8 @@
>  
>  #include "pcie-cadence.h"
>  
> +#define LINK_RETRAIN_TIMEOUT HZ
> +
>  static u64 bar_max_size[] = {
>  	[RP_BAR0] = _ULL(128 * SZ_2G),
>  	[RP_BAR1] = SZ_2G,
> @@ -77,6 +79,27 @@ static struct pci_ops cdns_pcie_host_ops = {
>  	.write		= pci_generic_config_write,
>  };
>  
> +static int cdns_pcie_host_training_complete(struct cdns_pcie *pcie)

This is kind of weird because it's named like a predicate, i.e., "this
function tells me whether link training is complete", but it returns
*zero* for success.

This is the opposite of j721e_pcie_link_up(), which returns "true"
when the link is up, so code like this reads naturally:

  if (pcie->ops->link_up(pcie))
    /* do something if the link is up */

> +{
> +	u32 pcie_cap_off = CDNS_PCIE_RP_CAP_OFFSET;
> +	unsigned long end_jiffies;
> +	u16 lnk_stat;
> +
> +	/* Wait for link training to complete. Exit after timeout. */
> +	end_jiffies = jiffies + LINK_RETRAIN_TIMEOUT;
> +	do {
> +		lnk_stat = cdns_pcie_rp_readw(pcie, pcie_cap_off + PCI_EXP_LNKSTA);
> +		if (!(lnk_stat & PCI_EXP_LNKSTA_LT))
> +			break;
> +		usleep_range(0, 1000);
> +	} while (time_before(jiffies, end_jiffies));
> +
> +	if (!(lnk_stat & PCI_EXP_LNKSTA_LT))
> +		return 0;
> +
> +	return -ETIMEDOUT;
> +}
> +
>  static int cdns_pcie_host_wait_for_link(struct cdns_pcie *pcie)
>  {
>  	struct device *dev = pcie->dev;
> @@ -118,6 +141,10 @@ static int cdns_pcie_retrain(struct cdns_pcie *pcie)
>  		cdns_pcie_rp_writew(pcie, pcie_cap_off + PCI_EXP_LNKCTL,
>  				    lnk_ctl);
>  
> +		ret = cdns_pcie_host_training_complete(pcie);
> +		if (ret)
> +			return ret;
> +
>  		ret = cdns_pcie_host_wait_for_link(pcie);

It seems a little clumsy that we wait for two things in succession:

  - cdns_pcie_host_training_complete() waits up to 1s for
    PCI_EXP_LNKSTA_LT to be cleared

  - cdns_pcie_host_wait_for_link() waits between .9s and 1s for
    LINK_UP_DL_COMPLETED on j721e (and not at all for other platforms)

dw_pcie_wait_for_link() is basically similar but has a single wait
loop around the dw_pcie_link_up() callback.  Several of those
callbacks check multiple things.  Can we do the same here?

Is the "host" in the cdns_pcie_host_wait_for_link() name necessary?
Maybe it could be cdns_pcie_wait_for_link() to be similar to
dw_pcie_wait_for_link()?  Or, if "host" is necessary, it could be
cdns_host_pcie_wait_for_link() so it matches the same
"pcie_wait_for_link" grep pattern as most of the others?

>  	}
>  	return ret;