Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
Date:   Thu, 30 Nov 2023 15:17:39 -0500
From:   Frank Li <Frank.li@nxp.com>
To:     Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Cc:     bhelgaas@google.com, imx@lists.linux.dev, kw@linux.com,
        linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org,
        linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,
        lpieralisi@kernel.org, minghuan.Lian@nxp.com, mingkai.hu@nxp.com,
        robh@kernel.org, roy.zang@nxp.com
Subject: Re: [PATCH v4 4/4] PCI: layerscape: Add suspend/resume for ls1043a
Message-ID: <ZWjt412xtyZWVjdL@lizhi-Precision-Tower-5810>
References: <20231129214412.327633-1-Frank.Li@nxp.com>
 <20231129214412.327633-5-Frank.Li@nxp.com>
 <20231130165100.GV3043@thinkpad>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20231130165100.GV3043@thinkpad>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Wk92cmJTeGxqNjBZTUIvMlprUjdQQjVocTR0UEpCOU9MWjZaQkFOeFhxaDN4?=
 =?utf-8?B?OTE0aEVTbTNBZ0FUQ1FYN21mR3Zuem1lZElZakhJMkc4dWdkSjhYYzc4T2cy?=
 =?utf-8?B?djhheVVZZDY5NTkwMGlySE9hSHlzckc4OUU5bzBWNmhuK3c5a3F3VDg4WnNs?=
 =?utf-8?B?dFFhYytOSnJOTERvZHE5SmdETHVTRzdMU2lKWHJ2dThFTlZURGxGamROY0Q2?=
 =?utf-8?B?OVVlN1FHdlJLeHN1Q1ZlUy84VnlKemhGOEh1QmNrVjB4SzU1b1RiWG1zY0JI?=
 =?utf-8?B?czNLMlZjNVRRNThpNlNBQVN4c2pqMjhBY2h5Slo3bTVnNHpUN202dmloOU8z?=
 =?utf-8?B?dmhEaS9uZTFqRnhDdzdXVTJWQWZRdGJEUkIray9GQnZ2dytsa3c2dDhmMW42?=
 =?utf-8?B?S2NyV05OU2lEMUZFM2dlbGZpVnBrK0dBeUJZTStscEdkb01mVkpqcWFhZ2k3?=
 =?utf-8?B?cUNsWU1GSFY5bXphaUVrYzhscXZibTJNYzJsQW9qVFovU2tFL3lKbUp4VThR?=
 =?utf-8?B?TzVJV1Z5Z0dOUks0eEpoRVVZb2lGem1sZTdHQUZKVkl3cHUvem92S0J1RlN4?=
 =?utf-8?B?UDI2bVcxK29pcHNDWGk0QjlSTDgzTWhFV1JlRmluS2hGQ0diYlZZZkJVc0Ux?=
 =?utf-8?B?RXBTSmNLYjI4TW5ReXlFUlFxT1dUNGhEZEZoQWYrWExMODdFamNnSHNOTnpG?=
 =?utf-8?B?OFNURFZNUm9xdVA4N3RCQTNDVnNxbVVKcUNvWjh6OTgxUS9QdURtTDdaZ0hP?=
 =?utf-8?B?WjM3ZzFrSHc2Zjc2ZE5BcnlVL2tBenM3Nzc3ZVA4SExrSDBqV29kTXJXaGYx?=
 =?utf-8?B?dGJVaFNNcDIzZk9MK1FFODdrRnlzU1J5RCtjdnJldlhlNU8yajVlZ3JGUEZk?=
 =?utf-8?B?bm12MldsdzJKM2wvR3R3V0pXWU45TTMrQUlyS3Y4bDk5MnBUaDhuZjdQcFln?=
 =?utf-8?B?QUViVnl4ZTh2aVN3Zk9DWXo2emNBRTB2WWQ5SkdBSm5YaXhBVkZmUkpVMkhj?=
 =?utf-8?B?eFV0bk9xYmJSY2tqMy82b29sVzMva1JpSiswSFA2Wjc4NjVLU2RNOE5uTE5R?=
 =?utf-8?B?UGUra2krcldRWFRJNVZqTktyVlYyVWlNMFo0Zng0dGE0aDFVQndkMmhDQWRP?=
 =?utf-8?B?bG5rbHo2QzkrZ1RvODQvL0FoQjZCci90NXIwZ25URjVibzVGbGRIeWkvZFdn?=
 =?utf-8?B?UkU3SXF3bXVXVWQ1UThpdEZDL2JueHJmQ2hNT1RBZ1F4czhWUUU1N1ZXQ3Zk?=
 =?utf-8?B?dDc3ekxWTWtuREtzRTVselI0a2JIYnhINU9IY3J5a2h0OGVGdkZET2hOVHQ3?=
 =?utf-8?B?SzFNSTF6d3ppN0NDV2V0Z3NjZ3B0MjJhWFYvSEkvM2E4ZlRUOHlsQktENmd1?=
 =?utf-8?B?Yk00bUFWYUR2NlNKNUYxM285UUhLem9RSWQ2OU9Hd05xSUZvbGtzdTJnd0Ja?=
 =?utf-8?B?dTdOZmdMb01jcmpBaFBkSWxIZCt5MzNxUHNLTUVUQUZreUpHenVFdmxwSmc1?=
 =?utf-8?B?RURUWWRCQzRsSlZjcHY5dCtFSFk0Mjk3bzZSSnVoWmdwN0NWRnJpWU8xTXBU?=
 =?utf-8?B?UVVVaHNYOGpjR3hTWEZoOEJvVEpicjFxcUpqZlYzMyszMFFobTYwZ3Faa1hI?=
 =?utf-8?B?Uy9mR1AvTzQ4ZUVQdm00T1JhQUFUSEJkNnVMTytGWEtTODVzemcweklYRnFw?=
 =?utf-8?B?T0dHTjQrQmJjY21Kcm0xUWdGN0JVZExiTHdhQmpqbjBtN1AxcmFjREFybCsr?=
 =?utf-8?B?YmJEbktwQUN1SUM3WFY3VHR6eHI0R1NWTVhLaFJFNEN3VXkySTVlaTZTdDVq?=
 =?utf-8?B?ZlZveDVKSWNXWW1MdW96MEt0dHRvQUdyOE1lcWZ0SmxHU2d6OVIwUW11QlJU?=
 =?utf-8?B?aTdFYjFuM2NVcko4cWErbmV5Rm1qc2FSVmIvSFNRMzRvQkVNYUlLeTFwWnZX?=
 =?utf-8?B?WHQ3Q3pGcllUTGhhR2E0aDl4alBDSVB3R055N0JPdkRKWVI5R1lqclBDemZx?=
 =?utf-8?B?Mk1PeGJrYTczNTdjWFVhQ3lkQUhZRUxaV1lFWjdoNTF3Z09OMGwwQWwwUS92?=
 =?utf-8?B?ZkJtK09yUGE5bDNIU3RwTlVieHdIT0RWUHQ2Vm5qVlRhOEZnYnVRWlpoQzNX?=
 =?utf-8?Q?JOa4=3D?=
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b23661c9-8a0d-4b5a-8fb4-08dbf1e16b10
X-MS-Exchange-CrossTenant-AuthSource: AM6PR04MB4838.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Nov 2023 20:17:47.3036
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: a94QdLmhZOBqXBW/9g/vIBLeicBcFf3kDqS0L29EGPIV7OXhEiU8u6lI09qFHQq3bb7zDKps/bCsckqCHVVFIg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM7PR04MB6838
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Thu, 30 Nov 2023 12:18:04 -0800 (PST)

On Thu, Nov 30, 2023 at 10:21:00PM +0530, Manivannan Sadhasivam wrote:
> On Wed, Nov 29, 2023 at 04:44:12PM -0500, Frank Li wrote:
> > In the suspend path, PME_Turn_Off message is sent to the endpoint to
> > transition the link to L2/L3_Ready state. In this SoC, there is no way to
> > check if the controller has received the PME_To_Ack from the endpoint or
> > not. So to be on the safer side, the driver just waits for
> > PCIE_PME_TO_L2_TIMEOUT_US before asserting the SoC specific PMXMTTURNOFF
> > bit to complete the PME_Turn_Off handshake. This link would then enter
> > L2/L3 state depending on the VAUX supply.
> > 
> > In the resume path, the link is brought back from L2 to L0 by doing a
> > software reset.
> > 
> 
> Same comment on the patch description as on patch 2/4.
> 
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> > 
> > Notes:
> >     Change from v3 to v4
> >     - Call scfg_pcie_send_turnoff_msg() shared with ls1021a
> >     - update commit message
> >     
> >     Change from v2 to v3
> >     - Remove ls_pcie_lut_readl(writel) function
> >     
> >     Change from v1 to v2
> >     - Update subject 'a' to 'A'
> > 
> >  drivers/pci/controller/dwc/pci-layerscape.c | 63 ++++++++++++++++++++-
> >  1 file changed, 62 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/pci/controller/dwc/pci-layerscape.c b/drivers/pci/controller/dwc/pci-layerscape.c
> > index 590e07bb27002..d39700b3afaaa 100644
> > --- a/drivers/pci/controller/dwc/pci-layerscape.c
> > +++ b/drivers/pci/controller/dwc/pci-layerscape.c
> > @@ -41,6 +41,15 @@
> >  #define SCFG_PEXSFTRSTCR	0x190
> >  #define PEXSR(idx)		BIT(idx)
> >  
> > +/* LS1043A PEX PME control register */
> > +#define SCFG_PEXPMECR		0x144
> > +#define PEXPME(idx)		BIT(31 - (idx) * 4)
> > +
> > +/* LS1043A PEX LUT debug register */
> > +#define LS_PCIE_LDBG	0x7fc
> > +#define LDBG_SR		BIT(30)
> > +#define LDBG_WE		BIT(31)
> > +
> >  #define PCIE_IATU_NUM		6
> >  
> >  struct ls_pcie_drvdata {
> > @@ -225,6 +234,45 @@ static int ls1021a_pcie_exit_from_l2(struct dw_pcie_rp *pp)
> >  	return scfg_pcie_exit_from_l2(pcie->scfg, SCFG_PEXSFTRSTCR, PEXSR(pcie->index));
> >  }
> >  
> > +static void ls1043a_pcie_send_turnoff_msg(struct dw_pcie_rp *pp)
> > +{
> > +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > +	struct ls_pcie *pcie = to_ls_pcie(pci);
> > +
> > +	scfg_pcie_send_turnoff_msg(pcie->scfg, SCFG_PEXPMECR, PEXPME(pcie->index));
> > +}
> > +
> > +static int ls1043a_pcie_exit_from_l2(struct dw_pcie_rp *pp)
> > +{
> > +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > +	struct ls_pcie *pcie = to_ls_pcie(pci);
> > +	u32 val;
> > +
> > +	/*
> > +	 * Only way let PEX module exit L2 is do a software reset.
> 
> Can you expand PEX? What is it used for?
> 
> Also if the reset is only for the PEX module, please use the same comment in
> both patches 2 and 4. Patch 2 doesn't mention PEX in the comment.

After read spec again, I think PEX is pci express. So it should software
reset controller. I don't know what exactly did in the chip. But without
below code, PCIe can't exit L2/L3.

Any harmful if dwc controller reset? Anyway these code works well with
intel network card.

Frank

> 
> - Mani
> 
> > +	 * LDBG_WE: allows the user to have write access to the PEXDBG[SR] for both setting and
> > +	 *	    clearing the soft reset on the PEX module.
> > +	 * LDBG_SR: When SR is set to 1, the PEX module enters soft reset.
> > +	 */
> > +	val = ls_pcie_pf_lut_readl(pcie, LS_PCIE_LDBG);
> > +	val |= LDBG_WE;
> > +	ls_pcie_pf_lut_writel(pcie, LS_PCIE_LDBG, val);
> > +
> > +	val = ls_pcie_pf_lut_readl(pcie, LS_PCIE_LDBG);
> > +	val |= LDBG_SR;
> > +	ls_pcie_pf_lut_writel(pcie, LS_PCIE_LDBG, val);
> > +
> > +	val = ls_pcie_pf_lut_readl(pcie, LS_PCIE_LDBG);
> > +	val &= ~LDBG_SR;
> > +	ls_pcie_pf_lut_writel(pcie, LS_PCIE_LDBG, val);
> > +
> > +	val = ls_pcie_pf_lut_readl(pcie, LS_PCIE_LDBG);
> > +	val &= ~LDBG_WE;
> > +	ls_pcie_pf_lut_writel(pcie, LS_PCIE_LDBG, val);
> > +
> > +	return 0;
> > +}
> > +
> >  static const struct dw_pcie_host_ops ls_pcie_host_ops = {
> >  	.host_init = ls_pcie_host_init,
> >  	.pme_turn_off = ls_pcie_send_turnoff_msg,
> > @@ -242,6 +290,19 @@ static const struct ls_pcie_drvdata ls1021a_drvdata = {
> >  	.exit_from_l2 = ls1021a_pcie_exit_from_l2,
> >  };
> >  
> > +static const struct dw_pcie_host_ops ls1043a_pcie_host_ops = {
> > +	.host_init = ls_pcie_host_init,
> > +	.pme_turn_off = ls1043a_pcie_send_turnoff_msg,
> > +};
> > +
> > +static const struct ls_pcie_drvdata ls1043a_drvdata = {
> > +	.pf_lut_off = 0x10000,
> > +	.pm_support = true,
> > +	.scfg_support = true,
> > +	.ops = &ls1043a_pcie_host_ops,
> > +	.exit_from_l2 = ls1043a_pcie_exit_from_l2,
> > +};
> > +
> >  static const struct ls_pcie_drvdata layerscape_drvdata = {
> >  	.pf_lut_off = 0xc0000,
> >  	.pm_support = true,
> > @@ -252,7 +313,7 @@ static const struct of_device_id ls_pcie_of_match[] = {
> >  	{ .compatible = "fsl,ls1012a-pcie", .data = &layerscape_drvdata },
> >  	{ .compatible = "fsl,ls1021a-pcie", .data = &ls1021a_drvdata },
> >  	{ .compatible = "fsl,ls1028a-pcie", .data = &layerscape_drvdata },
> > -	{ .compatible = "fsl,ls1043a-pcie", .data = &ls1021a_drvdata },
> > +	{ .compatible = "fsl,ls1043a-pcie", .data = &ls1043a_drvdata },
> >  	{ .compatible = "fsl,ls1046a-pcie", .data = &layerscape_drvdata },
> >  	{ .compatible = "fsl,ls2080a-pcie", .data = &layerscape_drvdata },
> >  	{ .compatible = "fsl,ls2085a-pcie", .data = &layerscape_drvdata },
> > -- 
> > 2.34.1
> > 
> 
> -- 
> மணிவண்ணன் சதாசிவம்