Received-SPF: pass (google.com: domain of linux-kernel+bounces-166860-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Thu, 2 May 2024 15:17:50 -0400
From: Frank Li <Frank.li@nxp.com>
To: Bjorn Helgaas <bhelgaas@google.com>, Jingoo Han <jingoohan1@gmail.com>,
	Gustavo Pimentel <gustavo.pimentel@synopsys.com>,
	Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>,
	Lorenzo Pieralisi <lpieralisi@kernel.org>,
	Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= <kw@linux.com>,
	Rob Herring <robh@kernel.org>,
	Krzysztof Kozlowski <krzysztof.kozlowski+dt@linaro.org>,
	Conor Dooley <conor+dt@kernel.org>, imx@lists.linux.dev
Cc: linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org,
	devicetree@vger.kernel.org,
	Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>,
	Serge Semin <fancer.lancer@gmail.com>
Subject: Re: [PATCH v8 0/5] PCI: dwc: Add common pme_turn_off message by
 using outbound iATU
Message-ID: <ZjPm3oSQQW6B8vm1@lizhi-Precision-Tower-5810>
References: <20240418-pme_msg-v8-0-a54265c39742@nxp.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240418-pme_msg-v8-0-a54265c39742@nxp.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?sNHKWL7UyohbT8OsS7xQTVazFmpfYPt8H9ycGhMyQA4MjGfa+O1RFqOwpbqd?=
 =?us-ascii?Q?L8sq/a4MntAN4//oLUv9Nz7QjZrHpq+SEyrnR7TKN6vmtVIXYvMJsatoyFz1?=
 =?us-ascii?Q?3mZlOLsUVACn7OLkeQ8tSb1ajI4PneC2ddyUm78W3Tnzwfqf9rHZLFfSfEi+?=
 =?us-ascii?Q?NoC3wYzKAWVg50f0sKm5qEu2qpkCfy/QXj2g8r/ypPGoKyFi3ys+0LF7B9uJ?=
 =?us-ascii?Q?WpBoQzikLJ7fExwCr8GQ/hIFQ25HV0wr0PKNxSMWJwWeNi9E5pKMgAOav3My?=
 =?us-ascii?Q?HZKXJq1HYjAq/H+2yfrle5S+VyfxsArU2g4j0k2cN0lwDM75JmZfu2DMGp0t?=
 =?us-ascii?Q?PCiC20PBwRj24/SPwA7fQliVtxK8eT69MGthi5vqaPKgYDXnVtrbTNN2F9ec?=
 =?us-ascii?Q?PS6UnTuMpLETtVqqWlRiBptzpftmkt9b8/B8wj2kdEELhHwAwRGsEwPHwrK9?=
 =?us-ascii?Q?GgUDLXwrgdOVN4MKIv1/gl7tzZjjoGyasBE5m2gQ+txagIhZ/DYpM/IkO8wb?=
 =?us-ascii?Q?cXWR45EVUIBT063vqq2nlfvBQ3uQETGc86/uX0F7qnDjCHd5oXKNnjCfoiif?=
 =?us-ascii?Q?WCDf5Iv8SPfUrNW3s+Yk0Wlp28vFMb8IKFc3wvQOlw+1xmiD5VQsZMdF5pgO?=
 =?us-ascii?Q?12fgpAZcBr6qDo4lBjXB1VTa8uY2j/4uQeEa5TbcFikP2OjLTmphvE26lP2V?=
 =?us-ascii?Q?8tDkA9h7kbj9ozGsusRHx9JbVNCktG5rO5EnvrokOpuR6pqkud4Stf/ukXpe?=
 =?us-ascii?Q?sbvg2ATZh6h9sMuD+ri4daJW22vS8dSlgxgZTmapNZG77gARIJERb5M8U2s7?=
 =?us-ascii?Q?K9+KYRPvlxWzeeCPPH7C9+NqB85U8jcoqqL/XbYM9CXzVhR6rIiMcrPbTbob?=
 =?us-ascii?Q?Cz8uT1wCVt/f+eU+q3HTnjYD5lIuD4Dhoe6S2Lso7dvCOyiPfKbSY3hcwFWg?=
 =?us-ascii?Q?Y+6LHnV1EhC4yf/Iw8X9IhWIuNZ7FBT49BJZZLpSwkCtvOkAthGQY5LBLZbc?=
 =?us-ascii?Q?nC7MkE7sjOuocxUqvsucTArjrN5vbjxRWUJXyoO0K5YCRP1cA18axUvq1Us9?=
 =?us-ascii?Q?/1R41spHCSEn0pvCWd/xtju/ofXDE1X1kbKxn/JoHS1ze42b3ZYxSWFLrb0e?=
 =?us-ascii?Q?zZ+q/ArxutUOLaR879KznWZy++cSLsZqQvUj185V2ZwN7I6MamFTepwhXE2g?=
 =?us-ascii?Q?mXVTW+eQOqG0EN6xf8sr94Up/NJTO6obWqiQItsBvZUmi0jLd5gZUfdcFX0G?=
 =?us-ascii?Q?StkArmrkX1BRhIcQ2UbmS4BrtQV92B8mIK29nB5QVwB59c7IeFCOTpQoHun2?=
 =?us-ascii?Q?hz75pgqVigN+RjyBCUKPi/kAOKr83dS0CxShnXtNJeGbEZjBU4vnNYeBDvgf?=
 =?us-ascii?Q?XtUpiGNVjXvPbIA2NTWAueLKce6A+P12QspcAOgbKPlepNnSZygpbi5vA33K?=
 =?us-ascii?Q?PC5oXrh4HOJ6Fzob9Y6st/T9aNs/0ylfkL9OpSQWlL+MPjKcltKi8Xxbg9Iu?=
 =?us-ascii?Q?JPrHkJLbTOBYhHmKmkuCK6u5IRUzQf0JZggIm08xzPlOuLQpKq/ZcMLwwTa+?=
 =?us-ascii?Q?pMYY9pTJiUWtX8yxB80=3D?=
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-Network-Message-Id: eb5d3846-ba4f-4d50-9722-08dc6adc945e
X-MS-Exchange-CrossTenant-AuthSource: DB9PR04MB9626.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 May 2024 19:17:59.5448
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: MYQXcAilcV8lwc/mhcb0yvrHpfsk3whW9FLfl2Ab7VyKWUMBHAnyB3MGh4xKMxLAbbbx6eELuv7RE0e2y7El8Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR04MB8229

On Thu, Apr 18, 2024 at 12:04:23PM -0400, Frank Li wrote:
> Involve an new and common mathod to send pme_turn_off() message. Previously
> pme_turn_off() implement by platform related special register to trigge    
> it.                                                                        
>                                                                            
> But Yoshihiro give good idea by using iATU to send out message. Previously 
> Yoshihiro provide patches to raise INTx message by dummy write to outbound 
> iATU.                                                                      
>                                                                            
> Use similar mathod to send out pme_turn_off message.                       
>                                                                            
> Previous two patches is picked from Yoshihiro' big patch serialise.        
>  PCI: dwc: Change arguments of dw_pcie_prog_outbound_atu()                 
>  PCI: Add INTx Mechanism Messages macros                                   
>                                                                            
> PCI: Add PME_TURN_OFF message macro                                        
> dt-bindings: PCI: dwc: Add 'msg" register region, Add "msg" region to use  
> to map PCI msg.                                                            
>                                                                            
> PCI: dwc: Add common pme_turn_off message method                           
> Using common pme_turn_off() message if platform have not define their.
> 
> Signed-off-by: Frank Li <Frank.Li@nxp.com>

Lorenzo:

	Could you please pick up these patches? All have been reviewed and
acked by Bjorn/Mani? If you have more comments, let me know.

Frank

> ---
> Changes in v8:
> - Total use devm_ manage msg_res. after read devres.c code to better
> understand release sequence. It use reverse order to release resource. So
> it is safe for msg_res.
> - Add Reviewed tag of Manivannan Sadhasivam in patch 4/5
> - Link to v7: https://lore.kernel.org/r/20240417-pme_msg-v7-0-9c6208cda90f@nxp.com
> 
> Changes in v7:
>   patch 4/5 add
>   /* Power Management Messages; PCIe r6.0, sec 2.2.8.2 */
>   patch 1/5 change comment to
>   /* Message Routing (r[2:0]); PCIe r6.0, sec 2.2.8 */
>   /* INTx Mechanism Messages; PCIe r6.0, sec 2.2.8.1 */
> 
>   order according to value.
>   Add Acked-by: Bjorn Helgaas <bhelgaas@google.com> in patch 4/5
> 
>   patch 5/5
>   Apply all mani's comments.
> 
> - Link to v6: https://lore.kernel.org/r/20240415-pme_msg-v6-0-56dad968ad3a@nxp.com
> 
> Changes in v6:
> - Move PCIE_MSG_CODE_PME_TURN_OFF ahead before INTx to keep the codes
> sorted.
> - Added reason why not alloc_resesource in commit messag.
> - Change title PCI: dwc: Add generic MSG TLP support for sending
> PME_Turn_Off when system suspend.
> - Rework commit message
>   "Instead of relying on the vendor specific implementations to send the
> PME_Turn_Off message, let's introduce a generic way of sending the message using
> the MSG TLP.
> 
> This is achieved by reserving a region for MSG TLP of size 'pci->region_align',
> at the end of the first IORESOURCE_MEM window of the host bridge. And then
> sending the PME_Turn_Off message during system suspend with the help of iATU.
> 
> It should be noted that this generic implementation is optional for the glue
> drivers and can be overridden by a custom 'pme_turn_off' callback.
> 
> - using devm_request_resource()
> - using mani's suggest comments
> 
> 	/*
> 	 * Allocate the resource for MSG TLP before programming the iATU
> 	 * outbound window in dw_pcie_setup_rc(). Since the allocation depends
> 	 * on the value of 'region_align', this has to be done after
> 	 * dw_pcie_iatu_detect().
> 	 */
> 
> - Add comments, about use_atu_msg, need glue driver set it.
> 
> - m -> mem in dw_pcie_pme_turn_off()
> 
> - remove redundant check() if (!atu.size) in dw_pcie_pme_turn_off().
> 
> - The reason can't adjust windows->end
> This windows already reported to pci system before this function. It is
> not good to modify window-res-end. It just add child resource like below.
> 
> windows is root resource, which will create may child when call
> request_resource.
>           bridge -> windows
> 		child1 -> msg
> 		child2 -> pci ep1
> 		child3 -> pci_ep2.
> 		...
> 
> Although you see whole bridge window, 'msg' already used and put under root
> resource,  new pci devices will never use 'msg' resource. 
> 
> If change windows->res->end here, I worry about it may broken resource
> tree.
> 
> Detail below thread:
> 
> - Link to v5: https://lore.kernel.org/r/20240319-pme_msg-v5-0-af9ffe57f432@nxp.com
> 
> Changes in v5:
> - Default disable allocate TLP message memory windows. If driver need use
> this feature, need set use_atu_msg = true before call dw_host_init().
> 
> - Link to v4: https://lore.kernel.org/r/20240213-pme_msg-v4-0-e2acd4d7a292@nxp.com
> 
> Changes in v4:
> - Remove dt-binding patch. Needn't change any dts file and binding doc.
>   Reserve a region at end of first IORESOURCE_MEM window by call
>   request_resource(). So PCIe stack will not use this reserve region to any
> PCIe devices.
>   I tested it by reserve at begin of IORESOURCE_MEM window. PCIe stack
> will skip it as expection.
> 
>   Fixed a issue, forget set iATU index when sent PME_turn_off.
> 
> - Link to v3: https://lore.kernel.org/r/20240202-pme_msg-v3-0-ff2af57a02ad@nxp.com
> 
> Changes in v3:
> - fix 'MSG"
> - Add pcie spec ref in head file
> - using function name dw_pci_pme_turn_off()
> - Using PCIE_ prefix macro
> - Link to v2: https://lore.kernel.org/r/20240201-pme_msg-v2-0-6767052fe6a4@nxp.com
> 
> Changes in v2:
>   - Add my sign off at PCI: dwc: Add outbound MSG TLPs support
>   - Add Bjorn review tag at  Add INTx Mechanism Messages macros
>   - using PME_Turn_Off match PCIe spec
>   - ref to pcie spec v6.1
>   - using section number.
> 
> - Link to v1: https://lore.kernel.org/r/20240130-pme_msg-v1-0-d52b0add5c7c@nxp.com
> 
> ---
> Frank Li (2):
>       PCI: Add PCIE_MSG_CODE_PME_TURN_OFF message macro
>       PCI: dwc: Add generic MSG TLP support for sending PME_Turn_Off when system suspend
> 
> Yoshihiro Shimoda (3):
>       PCI: Add INTx Mechanism Messages macros
>       PCI: dwc: Consolidate args of dw_pcie_prog_outbound_atu() into a structure
>       PCI: dwc: Add outbound MSG TLPs support
> 
>  drivers/pci/controller/dwc/pcie-designware-ep.c   |  21 ++--
>  drivers/pci/controller/dwc/pcie-designware-host.c | 145 +++++++++++++++++++---
>  drivers/pci/controller/dwc/pcie-designware.c      |  54 ++++----
>  drivers/pci/controller/dwc/pcie-designware.h      |  22 +++-
>  drivers/pci/pci.h                                 |  21 ++++
>  5 files changed, 199 insertions(+), 64 deletions(-)
> ---
> base-commit: ef6a734484e939cf67bc41c45bc936a78ea210f0
> change-id: 20240130-pme_msg-dd2d81ee9886
> 
> Best regards,
> ---
> Frank Li <Frank.Li@nxp.com>
>