Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   "Pankaj Dubey" <pankaj.dubey@samsung.com>
To:     "'Vidya Sagar'" <vidyas@nvidia.com>,
        "'Gustavo Pimentel'" <Gustavo.Pimentel@synopsys.com>,
        "'Andrew Murray'" <andrew.murray@arm.com>
Cc:     <linux-pci@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <jingoohan1@gmail.com>, <lorenzo.pieralisi@arm.com>,
        <bhelgaas@google.com>, "'Anvesh Salveru'" <anvesh.s@samsung.com>
In-Reply-To: <7ad2b603-49ce-e955-58c4-fba1fb5ca6c8@nvidia.com>
Subject: RE: [PATCH v2] PCI: dwc: Add support to add GEN3 related
 equalization quirks
Date:   Tue, 24 Sep 2019 14:58:26 +0530
Message-ID: <000001d572ba$6d3040a0$4790c1e0$@samsung.com>
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Thread-Index: AQKZmfmIjaX5BaBnMA3N+7vtT2Qq7gGeUxCAAtEE+g8BbnK8pAGPYjqeAXaY7iIC0ICEfKVUm0FA
Content-Language: en-us
Content-Type: text/plain; charset="utf-8"
CMS-TYPE: 105P
References: <CGME20190913104018epcas5p3d93265a6786dc2b7b8a7d3231bfe9c14@epcas5p3.samsung.com>
        <1568371190-14590-1-git-send-email-pankaj.dubey@samsung.com>
        <20190916101543.GM9720@e119886-lin.cambridge.arm.com>
        <00a401d56c7e$cf3abd30$6db03790$@samsung.com>
        <20190916122400.GO9720@e119886-lin.cambridge.arm.com>
        <DM6PR12MB4010AE07CC6F1CC60A715EE4DA8C0@DM6PR12MB4010.namprd12.prod.outlook.com>
        <7ad2b603-49ce-e955-58c4-fba1fb5ca6c8@nvidia.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> -----Original Message-----
> From: Vidya Sagar <vidyas=40nvidia.com>
> Sent: Thursday, September 19, 2019 4:54 PM
> Subject: Re: =5BPATCH v2=5D PCI: dwc: Add support to add GEN3 related equ=
alization
> quirks
>=20
> On 9/16/2019 6:22 PM, Gustavo Pimentel wrote:
> > On Mon, Sep 16, 2019 at 13:24:1, Andrew Murray
> <andrew.murray=40arm.com>
> > wrote:
> >
> >> On Mon, Sep 16, 2019 at 04:36:33PM +0530, Pankaj Dubey wrote:
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Andrew Murray <andrew.murray=40arm.com>
> >>>> Sent: Monday, September 16, 2019 3:46 PM
> >>>> To: Pankaj Dubey <pankaj.dubey=40samsung.com>
> >>>> Cc: linux-pci=40vger.kernel.org; linux-kernel=40vger.kernel.org;
> >>>> jingoohan1=40gmail.com; gustavo.pimentel=40synopsys.com;
> >>>> lorenzo.pieralisi=40arm.com; bhelgaas=40google.com; Anvesh Salveru
> >>>> <anvesh.s=40samsung.com>
> >>>> Subject: Re: =5BPATCH v2=5D PCI: dwc: Add support to add GEN3 relate=
d
> >>> equalization
> >>>> quirks
> >>>>
> >>>> On Fri, Sep 13, 2019 at 04:09:50PM +0530, Pankaj Dubey wrote:
> >>>>> From: Anvesh Salveru <anvesh.s=40samsung.com>
> >>>>>
> >>>>> In some platforms, PCIe PHY may have issues which will prevent
> >>>>> linkup to happen in GEN3 or higher speed. In case equalization
> >>>>> fails, link will fallback to GEN1.
> >>>>>
> >>>>> DesignWare controller gives flexibility to disable GEN3
> >>>>> equalization completely or only phase 2 and 3 of equalization.
> >>>>>
> >>>>> This patch enables the DesignWare driver to disable the PCIe GEN3
> >>>>> equalization by enabling one of the following quirks:
> >>>>>   - DWC_EQUALIZATION_DISABLE: To disable GEN3 equalization all
> >>>>> phases
> I don't think Gen-3 equalization can be skipped altogether.
> PCIe Spec Rev 4.0 Ver 1.0 in Section-4.2.3 has the following statement.
>=20
> =22All the Lanes that are associated with the LTSSM (i.e., those Lanes th=
at are
> currently operational or may be operational in the future due to Link
> Upconfigure) must participate in the Equalization procedure=22
>=20
> and in Section-4.2.6.4.2.1.1 it says
> =22Note: A transition to Recovery.RcvrLock might be used in the case wher=
e the
> Downstream Port determines that Phase 2 and Phase 3 are not needed based =
on
> the platform and channel characteristics.=22
>=20
> Based on the above statements, I think it is Ok to skip only Phases 2&3 o=
f
> equalization but not 0&1.
> I even checked with our hardware engineers and it seems
> DWC_EQUALIZATION_DISABLE is present only for debugging purpose in
> hardware simulations and shouldn't be used on real silicon otherwise it s=
eems.
>=20

In DesignWare manual we don't see any comment that this feature is for debu=
gging purpose only.
Even if it is meant for debugging purpose, if for some reason in an SoC, Ge=
n3/4 linkup is failing due to equalization, and if disabling equalization i=
s helping then IMO it is OK to do it.=20
Just to re-confirm we tested one of the NVMe device on Jatson AGX Xavier RC=
 with equalization disabled. We do see linkup works well in GEN3. As we hav=
e added this feature as a platform-quirk so only platforms that required th=
is feature can enable it.

Snippet of lspci (from Jatson AGX Xavier RC) is given below, showing EQ is =
completely disabled and GEN3 linkup
-----
0005:01:00.0 Non-Volatile memory controller: Lite-On Technology Corporation=
 Device 21f1 (rev 01) (prog-if 02 =5BNVM Express=5D)
        Subsystem: Marvell Technology Group Ltd. Device 1093
         <snip>
                LnkCap: Port =230, Speed 8GT/s, Width x4, ASPM L1, Exit Lat=
ency L0s <512ns, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLAct=
ive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LT=
R+, OBFF Via message
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR=
+, OBFF Disabled
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDi=
s-
                         Transmit Margin: Normal Operating Range, EnterModi=
fiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationCompl=
ete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqua=
lizationRequest-
-----
> - Vidya Sagar
>=20
>=20
> >>>>>   - DWC_EQ_PHASE_2_3_DISABLE: To disable GEN3 equalization phase 2
> >>>>> & 3
> >>>>>
> >>>>> Platform drivers can set these quirks via =22quirk=22 variable of =
=22dw_pcie=22
> >>>>> struct.
> >>>>>
> >>>>> Signed-off-by: Anvesh Salveru <anvesh.s=40samsung.com>
> >>>>> Signed-off-by: Pankaj Dubey <pankaj.dubey=40samsung.com>
> >>>>> ---
> >>>>> Patchset v1 can be found at:
> >>>>>   - 1/2: https://urldefense.proofpoint.com/v2/url?u=3Dhttps-
> 3A__lkml.org_lkml_2019_9_10_443&d=3DDwIBAg&c=3DDPL6_X_6JkXFx7AXWqB0tg
> &r=3DbkWxpLoW-f-
> E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=3DMtEKKeJsQvi2UM1eSZUv2vPLLxrYU0aI1
> Ry4ICIDaiQ&s=3Ds_nPmMNbQFswYRxQgBkeg4H9J_0FEtzRE-0AruC5WI4&e=3D
> >>>>>   - 2/2:
> >>>>> https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__lkml.org_lkm=
l
> >>>>>
> _2019_9_10_444&d=3DDwIBAg&c=3DDPL6_X_6JkXFx7AXWqB0tg&r=3DbkWxpLoW-f-
> E3Ed
> >>>>>
> iDCCa0_h0PicsViasSlvIpzZvPxs&m=3DMtEKKeJsQvi2UM1eSZUv2vPLLxrYU0aI1Ry
> >>>>> 4ICIDaiQ&s=3DkkfdwcX6bYcLrnJSgw_GcMMGAjnDTMtN2v6svWuANpk&e=3D
> >>>>>
> >>>>> Changes w.r.t v1:
> >>>>>   - Squashed two patches from v1 into one as suggested by Gustavo
> >>>>>   - Addressed review comments from Andrew
> >>>>>
> >>>>>   drivers/pci/controller/dwc/pcie-designware.c =7C 12 ++++++++++++
> >>>>> drivers/pci/controller/dwc/pcie-designware.h =7C  9 +++++++++
> >>>>>   2 files changed, 21 insertions(+)
> >>>>>
> >>>>> diff --git a/drivers/pci/controller/dwc/pcie-designware.c
> >>>>> b/drivers/pci/controller/dwc/pcie-designware.c
> >>>>> index 7d25102..97fb18d 100644
> >>>>> --- a/drivers/pci/controller/dwc/pcie-designware.c
> >>>>> +++ b/drivers/pci/controller/dwc/pcie-designware.c
> >>>>> =40=40 -466,4 +466,16 =40=40 void dw_pcie_setup(struct dw_pcie *pci=
)
> >>>>>   		break;
> >>>>>   	=7D
> >>>>>   	dw_pcie_writel_dbi(pci, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
> >>>>> +
> >>>>> +	if (pci->quirk & DWC_EQUALIZATION_DISABLE) =7B
> >>>>> +		val =3D dw_pcie_readl_dbi(pci, PCIE_PORT_GEN3_RELATED);
> >>>>> +		val =7C=3D PORT_LOGIC_GEN3_EQ_DISABLE;
> >>>>> +		dw_pcie_writel_dbi(pci, PCIE_PORT_GEN3_RELATED, val);
> >>>>> +	=7D
> >>>>> +
> >>>>> +	if (pci->quirk & DWC_EQ_PHASE_2_3_DISABLE) =7B
> >>>>> +		val =3D dw_pcie_readl_dbi(pci, PCIE_PORT_GEN3_RELATED);
> >>>>> +		val =7C=3D PORT_LOGIC_GEN3_EQ_PHASE_2_3_DISABLE;
> >>>>> +		dw_pcie_writel_dbi(pci, PCIE_PORT_GEN3_RELATED, val);
> >>>>> +	=7D
> >>>>>   =7D
> >>>>> diff --git a/drivers/pci/controller/dwc/pcie-designware.h
> >>>>> b/drivers/pci/controller/dwc/pcie-designware.h
> >>>>> index ffed084..e428b62 100644
> >>>>> --- a/drivers/pci/controller/dwc/pcie-designware.h
> >>>>> +++ b/drivers/pci/controller/dwc/pcie-designware.h
> >>>>> =40=40 -29,6 +29,10 =40=40
> >>>>>   =23define LINK_WAIT_MAX_IATU_RETRIES	5
> >>>>>   =23define LINK_WAIT_IATU			9
> >>>>>
> >>>>> +/* Parameters for GEN3 related quirks */
> >>>>> +=23define DWC_EQUALIZATION_DISABLE	BIT(1)
> >>>>> +=23define DWC_EQ_PHASE_2_3_DISABLE	BIT(2)
> >>>>> +
> >>>>>   /* Synopsys-specific PCIe configuration registers */
> >>>>>   =23define PCIE_PORT_LINK_CONTROL		0x710
> >>>>>   =23define PORT_LINK_MODE_MASK		GENMASK(21, 16)
> >>>>> =40=40 -60,6 +64,10 =40=40
> >>>>>   =23define PCIE_MSI_INTR0_MASK		0x82C
> >>>>>   =23define PCIE_MSI_INTR0_STATUS		0x830
> >>>>>
> >>>>> +=23define PCIE_PORT_GEN3_RELATED		0x890
> >>>>
> >>>> I hadn't noticed this in the previous version - what is the proper
> >>>> name
> >>> for this
> >>>> register? Does it end in _RELATED?
> >>>
> >>> As per SNPS databook the name of the register is =22GEN3_RELATED_OFF=
=22.
> >>> It is port logic register so, to keep similarity with other port
> >>> logic registers in this file we named it as =22PCIE_PORT_GEN3_RELATED=
=22.
> >>
> >> OK.
> >>
> >> Reviewed-by: Andrew Murray <andrew.murray=40arm.com>
> >>
> >> Also is the SNPS databook publicly available? I'd be interested in
> >> reading it.
> >
> > The databook isn't openly available, sorry.
> >
> > Gustavo
> >
> >>
> >> Thanks,
> >>
> >> Andrew Murray
> >>
> >>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Andrew Murray
> >>>>
> >>>>> +=23define PORT_LOGIC_GEN3_EQ_PHASE_2_3_DISABLE	BIT(9)
> >>>>> +=23define PORT_LOGIC_GEN3_EQ_DISABLE		BIT(16)
> >>>>> +
> >>>>>   =23define PCIE_ATU_VIEWPORT		0x900
> >>>>>   =23define PCIE_ATU_REGION_INBOUND		BIT(31)
> >>>>>   =23define PCIE_ATU_REGION_OUTBOUND	0
> >>>>> =40=40 -244,6 +252,7 =40=40 struct dw_pcie =7B
> >>>>>   	struct dw_pcie_ep	ep;
> >>>>>   	const struct dw_pcie_ops *ops;
> >>>>>   	unsigned int		version;
> >>>>> +	unsigned int		quirk;
> >>>>>   =7D;
> >>>>>
> >>>>>   =23define to_dw_pcie_from_pp(port) container_of((port), struct
> >>>>> dw_pcie,
> >>>>> pp)
> >>>>> --
> >>>>> 2.7.4
> >>>>>
> >>>
> >
> >