Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6;
Date:   Mon, 16 Oct 2023 16:18:36 -0400
From:   Frank Li <Frank.li@nxp.com>
To:     Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Cc:     Minghuan Lian <minghuan.Lian@nxp.com>,
        Mingkai Hu <mingkai.hu@nxp.com>, Roy Zang <roy.zang@nxp.com>,
        Lorenzo Pieralisi <lpieralisi@kernel.org>,
        Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= <kw@linux.com>,
        Rob Herring <robh@kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        "open list:PCI DRIVER FOR FREESCALE LAYERSCAPE" 
        <linuxppc-dev@lists.ozlabs.org>,
        "open list:PCI DRIVER FOR FREESCALE LAYERSCAPE" 
        <linux-pci@vger.kernel.org>,
        "moderated list:PCI DRIVER FOR FREESCALE LAYERSCAPE" 
        <linux-arm-kernel@lists.infradead.org>,
        open list <linux-kernel@vger.kernel.org>, imx@lists.linux.dev
Subject: Re: [PATCH 2/3] PCI: layerscape: add suspend/resume for ls1021a
Message-ID: <ZS2anCpWakttzaAu@lizhi-Precision-Tower-5810>
References: <20230915184306.2374670-1-Frank.Li@nxp.com>
 <20230915184306.2374670-2-Frank.Li@nxp.com>
 <20231016165824.GF39962@thinkpad>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20231016165824.GF39962@thinkpad>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NVNFdTJhVmZueWZLbG9RWTJyQzJuZ1Q0KzFTdzExc1VucVFIMCtlY3NWZlYx?=
 =?utf-8?B?eTlSb09NL1RjSjBVME5vNjFNNFUxc0FPS3ovd3hHNi8vM2pLdDZnc1pQOGJv?=
 =?utf-8?B?NDdNK215a1VnUzRuMlI5YnV4RnBnUEJXUmJobWNobHd2WVc2azZUZHdETmlW?=
 =?utf-8?B?ZTNSMlpEQWhWNzdWMWp3M3FkTW03RzR3Z2ErMEFCanlUMU9JNU1PSzlpVzND?=
 =?utf-8?B?RjhiQ3VXVktZRDl3Mld5UWJhdU1rdFBnZDBPeUl1aVVQNU53am9NaVVKOHBo?=
 =?utf-8?B?SmpUbkFHSXRuay9BVjRqa290em5zSCtCMzRvbTlWWWxHV3NOOXNiUWE0bjB5?=
 =?utf-8?B?bFlBVGpyZmd2NGtjd3U2SFJxRzM1alU2OUdlYVJUd1Y3T0dzanAxREhyZUph?=
 =?utf-8?B?Rmo1YVVJQ0JnY3ZSZUJDNHI1WkxlWW9vclNydXR4SXJ6aUd3cHBGejEvWmJM?=
 =?utf-8?B?QXMvdGJKenU5TlNRNUJPTE1VamZNV1QzQTZYSVJMeEhWTXMrNHYwd2d6K0Fv?=
 =?utf-8?B?L21VSGxUK3JpdzZDd3M0emE4SndqWkp2NlhyQUtVcnRHeE5kcElDam9TMDFU?=
 =?utf-8?B?UzRSUFdGcGpVSTdJREFlbFNQSWxDUGNLNnY1MjVuM1RJUFlvcUxVL0NPNmgr?=
 =?utf-8?B?RHhEQnc5OXp0R25yRDNPeFd4YVhmQWtOTUpRTENKVW1qRFA4SisvY3JuTUxz?=
 =?utf-8?B?bUNFMllBT0t3L3NtcVlCQ0hKOUMzYUhycGM3VzJ1dERrZUtxV2kxTDFZN0F0?=
 =?utf-8?B?NUM4UHMvalo4eGFHVUF0YitmWldPWnpjYi8rZlNRUHh0UXFXM2VHYTJGSWhO?=
 =?utf-8?B?M2JrdldRK3d3SlpOOHBYWmM5eXdpMFN6em42dlUrWkp5Vno5ZS9SY21WK0pu?=
 =?utf-8?B?ZW5ScmczdlYrUXc1SkZhZDJlOEdiYU94UENUY1EwK0JjcjA4UGlWMTBFbFAr?=
 =?utf-8?B?aS8zVzJnT0x4MXZzWEJGNnFrTEhEOW5nZTVUbzBEbGZLYjhVa21ZU0wreWdI?=
 =?utf-8?B?a0RnaVZQZjhNYjdDYW12WXZzbkNlS0E4Z0U4TjIvQzRlVWF0RDhhazFvUHNt?=
 =?utf-8?B?Z0NYOU1BUWFrNXR6NGJVSE5nMjFqcUllcWw5Wk1pQk5rcHNUem1TR28yRUdm?=
 =?utf-8?B?a0VsNXZFT3NuNU01UXllL2VTMlJXdmQ4WjFBeEFVeXBuWkhHaTF2YnBkeWl0?=
 =?utf-8?B?SWVnbVJOVEowcUNhZFh6dEIvVzROV3BWdTQ2VTFRZ1E5amVrRlpBWWNlR0VG?=
 =?utf-8?B?YmY3WVZpdEQ0N2QyRkY3Nmg0MHFIOHlRbGwvSkxnaWRaR2kvVkZoNkt3Z1JH?=
 =?utf-8?B?MnVxN0Y2OTBDaWk2UkNSeThLeE9YTlNjdEo5amkzbUxhYVZJYmRMODR3enhD?=
 =?utf-8?B?SnRtcDB6ZmNSRVBSN2Mwc2xtTkF4dFlSdTJhd1ZjQ2RpWmRUVFp4R09NZXRq?=
 =?utf-8?B?NFh5MXQ3OE81MTFLRUc2MFNGNFBTMW1zYTNndGZBWnorb3RYWVZXMVY5Rjdx?=
 =?utf-8?B?aUtiWGw3bnMrMzZxaFowZnB1SGt1dEczWi9BUVM5ZUZGRjVhcFRnQmEzdVBx?=
 =?utf-8?B?Y0dLREJjanFUdTk2Y1pVTFNBNll4eDUzcWkxcDFXL2FCbE9aT1VadkJMeEJO?=
 =?utf-8?B?a0hHRUl2bHdnOGFYTDY5bzlkR1NKNFlid0xKbHdud252MWVBd2VaUitEQUtI?=
 =?utf-8?B?d3liVlVqR1lINHNuOGxGcDBCSnphZk9WSDZnTlh3cHJNQWkyQ040WUNjbUdr?=
 =?utf-8?B?cW1GWWxSUUNqYnh0QkR4bXowS1k3U0c0eGZYekQzRko2NEcya0VrQWhRdTNn?=
 =?utf-8?B?Tmd3clVORVk3emJtcDR4aUV2bk9xYlYrOUtiNE0xaDJqNHl2UGRyendhY3di?=
 =?utf-8?B?blh2L01QSUdPRThvbVBldnNNQmJnWFcyTUhwMmZ5NDQvd1JNNzVLSlQ3Ymt0?=
 =?utf-8?B?YlRvMkFmaGdOc0xnayt3ZkNqYVhtV2pUT1lqRXVrb1B6WnlCeTQrQ3F3UjFw?=
 =?utf-8?B?MTVoRGx6OEg1L21yK3E1SUxjeEtkVzR2ODZPTURZdGV4TGxBSnQwZ3ZIZjAv?=
 =?utf-8?B?MzBlbFhuOWFSSndsVU81aTlTOFM5VTNHWDBlM2pqWjcydDkvcmRiZ28yM3A2?=
 =?utf-8?Q?91hE=3D?=
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2609b533-3902-40b9-2edf-08dbce8518ba
X-MS-Exchange-CrossTenant-AuthSource: AM6PR04MB4838.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Oct 2023 20:18:44.5366
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: SU4YDDpEsOy9Ro1T7xWrT5M8J9C7Nw0oxiqascO0ojpOXoZNo8sSFyuj77x118k09eQG6GymBL7bp09c4xK4AA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR04MB9389
X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 16 Oct 2023 13:19:08 -0700 (PDT)

On Mon, Oct 16, 2023 at 10:28:24PM +0530, Manivannan Sadhasivam wrote:
> On Fri, Sep 15, 2023 at 02:43:05PM -0400, Frank Li wrote:
> > ls1021a add suspend/resume support.
> > 
> 
> Please add what the driver is doing during suspend/resume.
> 
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> >  drivers/pci/controller/dwc/pci-layerscape.c | 88 ++++++++++++++++++++-
> >  1 file changed, 87 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/pci/controller/dwc/pci-layerscape.c b/drivers/pci/controller/dwc/pci-layerscape.c
> > index 20c48c06e2248..bc5a8ff1a26ce 100644
> > --- a/drivers/pci/controller/dwc/pci-layerscape.c
> > +++ b/drivers/pci/controller/dwc/pci-layerscape.c
> > @@ -35,6 +35,12 @@
> >  #define PF_MCR_PTOMR		BIT(0)
> >  #define PF_MCR_EXL2S		BIT(1)
> >  
> > +/* LS1021A PEXn PM Write Control Register */
> > +#define SCFG_PEXPMWRCR(idx)	(0x5c + (idx) * 0x64)
> > +#define PMXMTTURNOFF		BIT(31)
> > +#define SCFG_PEXSFTRSTCR	0x190
> > +#define PEXSR(idx)		BIT(idx)
> > +
> >  #define PCIE_IATU_NUM		6
> >  
> >  struct ls_pcie_drvdata {
> > @@ -48,6 +54,8 @@ struct ls_pcie {
> >  	struct dw_pcie *pci;
> >  	const struct ls_pcie_drvdata *drvdata;
> >  	void __iomem *pf_base;
> > +	struct regmap *scfg;
> > +	int index;
> >  	bool big_endian;
> >  };
> >  
> > @@ -170,13 +178,91 @@ static int ls_pcie_host_init(struct dw_pcie_rp *pp)
> >  	return 0;
> >  }
> >  
> > +static void ls1021a_pcie_send_turnoff_msg(struct dw_pcie_rp *pp)
> > +{
> > +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > +	struct ls_pcie *pcie = to_ls_pcie(pci);
> > +	u32 val;
> > +
> > +	if (!pcie->scfg) {
> 
> Can this ever happen?
> 
> > +		dev_dbg(pcie->pci->dev, "SYSCFG is NULL\n");
> > +		return;
> > +	}
> > +
> > +	/* Send Turn_off message */
> 
> "Send PME_Turn_Off message"
> 
> > +	regmap_read(pcie->scfg, SCFG_PEXPMWRCR(pcie->index), &val);
> > +	val |= PMXMTTURNOFF;
> > +	regmap_write(pcie->scfg, SCFG_PEXPMWRCR(pcie->index), val);
> > +
> > +	/* There are not register to check ACK, so wait PCIE_PME_TO_L2_TIMEOUT_US */
> 
> "There is no specific register to check for PME_To_Ack from endpoint. So on the
> safe side, wait for PCIE_PME_TO_L2_TIMEOUT_US."
> 
> > +	mdelay(PCIE_PME_TO_L2_TIMEOUT_US/1000);
> > +
> > +	/* Clear Turn_off message */
> 
> "PME_Turn_off". But I'm not sure if this is really required. Are you doing it
> because the layerspace hw implements the PME_Turn_Off bit as "level triggered"?

I am not sure how hardware implement this. But reference manual said:
 
PMXMTTURNOFF:
Generate PM turnoff message for power management of PCI Express controllers.
This bit should be cleared by software.
0 Clear PM turnoff (default)
1 Trigger PM turnoff

Frank

> 
> > +	regmap_read(pcie->scfg, SCFG_PEXPMWRCR(pcie->index), &val);
> > +	val &= ~PMXMTTURNOFF;
> > +	regmap_write(pcie->scfg, SCFG_PEXPMWRCR(pcie->index), val);
> > +}
> > +
> > +static void ls1021a_pcie_exit_from_l2(struct dw_pcie_rp *pp)
> > +{
> > +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > +	struct ls_pcie *pcie = to_ls_pcie(pci);
> > +	u32 val;
> > +
> 
> A comment here would be good.
> 
> > +	regmap_read(pcie->scfg, SCFG_PEXSFTRSTCR, &val);
> > +	val |= PEXSR(pcie->index);
> > +	regmap_write(pcie->scfg, SCFG_PEXSFTRSTCR, val);
> > +
> > +	regmap_read(pcie->scfg, SCFG_PEXSFTRSTCR, &val);
> > +	val &= ~PEXSR(pcie->index);
> > +	regmap_write(pcie->scfg, SCFG_PEXSFTRSTCR, val);
> > +}
> > +
> > +static int ls1021a_pcie_host_init(struct dw_pcie_rp *pp)
> > +{
> > +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > +	struct ls_pcie *pcie = to_ls_pcie(pci);
> > +	struct device *dev = pcie->pci->dev;
> > +	u32 index[2];
> > +	int ret;
> > +
> > +	ret = ls_pcie_host_init(pp);
> > +	if (ret)
> > +		return ret;
> > +
> > +	pcie->scfg = syscon_regmap_lookup_by_phandle(dev->of_node, "fsl,pcie-scfg");
> > +	if (IS_ERR(pcie->scfg)) {
> > +		ret = PTR_ERR(pcie->scfg);
> > +		dev_err(dev, "No syscfg phandle specified\n");
> > +		pcie->scfg = NULL;
> > +		return ret;
> > +	}
> > +
> > +	ret = of_property_read_u32_array(dev->of_node, "fsl,pcie-scfg", index, 2);
> > +	if (ret) {
> > +		pcie->scfg = NULL;
> > +		return ret;
> > +	}
> > +
> > +	pcie->index = index[1];
> > +
> 
> The above syscon parsing could be done conditionally during probe itself. There
> is no need to do it during host_init() time.
> 
> - Mani
> 
> > +	return ret;
> > +}
> > +
> >  static const struct dw_pcie_host_ops ls_pcie_host_ops = {
> >  	.host_init = ls_pcie_host_init,
> >  	.pme_turn_off = ls_pcie_send_turnoff_msg,
> >  };
> >  
> > +static const struct dw_pcie_host_ops ls1021a_pcie_host_ops = {
> > +	.host_init = ls1021a_pcie_host_init,
> > +	.pme_turn_off = ls1021a_pcie_send_turnoff_msg,
> > +};
> > +
> >  static const struct ls_pcie_drvdata ls1021a_drvdata = {
> > -	.pm_support = false,
> > +	.pm_support = true,
> > +	.ops = &ls1021a_pcie_host_ops,
> > +	.exit_from_l2 = ls1021a_pcie_exit_from_l2,
> >  };
> >  
> >  static const struct ls_pcie_drvdata layerscape_drvdata = {
> > -- 
> > 2.34.1
> > 
> 
> -- 
> மணிவண்ணன் சதாசிவம்