Received-SPF: pass (google.com: domain of linux-kernel+bounces-56746-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Wed, 7 Feb 2024 11:02:02 -0500
From: Frank Li <Frank.li@nxp.com>
To: Serge Semin <fancer.lancer@gmail.com>
Cc: Rob Herring <robh@kernel.org>, Bjorn Helgaas <bhelgaas@google.com>,
	Jingoo Han <jingoohan1@gmail.com>,
	Gustavo Pimentel <gustavo.pimentel@synopsys.com>,
	Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>,
	Lorenzo Pieralisi <lpieralisi@kernel.org>,
	Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= <kw@linux.com>,
	Krzysztof Kozlowski <krzysztof.kozlowski+dt@linaro.org>,
	Conor Dooley <conor+dt@kernel.org>, imx@lists.linux.dev,
	linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org,
	devicetree@vger.kernel.org
Subject: Re: [PATCH v3 5/6] dt-bindings: PCI: dwc: Add 'msg' register region
Message-ID: <ZcOpehO3rzCfAwXf@lizhi-Precision-Tower-5810>
References: <20240202-pme_msg-v3-0-ff2af57a02ad@nxp.com>
 <20240202-pme_msg-v3-5-ff2af57a02ad@nxp.com>
 <eg7wrjp5ebz43g37fvebr44nwkoh4rptbtyu76nalbmgbbnqke@4zugpgwesyqd>
 <20240205183048.GA3818249-robh@kernel.org>
 <ZcEzYdZKotBJlR5i@lizhi-Precision-Tower-5810>
 <ZcK2/tmLG9O7CBEH@lizhi-Precision-Tower-5810>
 <luk5hswq4wnk5p7axml73qih35hio3y3pfnklctbn6rwres62s@mumnvygjh5ch>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <luk5hswq4wnk5p7axml73qih35hio3y3pfnklctbn6rwres62s@mumnvygjh5ch>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?Ytpr4x+hmBtDTWkwW3JpH7oOP8yDUHCIYYFp/5+cnKeR6aQXvK6nO9616hEh?=
 =?us-ascii?Q?8noI7le4F8DUKZ8C0tpGimFdlPRV98lUdfQE54hmniaj0rJwRxrUyrPfuLuv?=
 =?us-ascii?Q?F1a9eADGDou0pAvpmhPM/YKCFSZFjRQlnLGGmdKxOGQIcmOk3u6hQlVLIlAm?=
 =?us-ascii?Q?RQU5x0lUUMKEgScKBfFn5eGaiM2BFqQInZCDKhPbELYJYZfteyH2VW//fsIw?=
 =?us-ascii?Q?UTiKTFazdSKQo8gmdKE2K7SSN8+8WJv3zXeJ7u15UD1V1rVlnDV+9T9MhI17?=
 =?us-ascii?Q?yx8+QiPVO1bzcjsasNVgdZpgrkCPzV/2pQutApAZyvbSuATdGegiE/OC8Niv?=
 =?us-ascii?Q?qv/GckD2N/rJtDyNw0+7QbkGO33hyfSMj41gwo+MYfpgFTalPaT2KtOQJT65?=
 =?us-ascii?Q?JtqlsqlyiEBzu/1fOIf71vV8sgPTQCBGOIQwgh8qfxeX+29cQ3d3y+ceXXsT?=
 =?us-ascii?Q?pwWc7PbnnrLcc2XVp1Ro+lcmd3E+kEijGsMjQKsmzVCbeHgMCMs718UrcB0/?=
 =?us-ascii?Q?I8r6pW0t+48iyfz+CdL4eMlpVu2Q5gjAJU1QxGiereuVjh7+sZjO8PNljmjn?=
 =?us-ascii?Q?4WdExWakEVtSRiIAKRfsXLeKmxXP3XDZ5U541hZ8sv/bMrzl81/qpuLSwPlL?=
 =?us-ascii?Q?AX23C1qnq0ManD+df1XrLqQJ9flSotvSsDYgpGmxlL5sPzoP7sYD0h837aIp?=
 =?us-ascii?Q?rEkVrQCP/u7XJg/PGI1Ndg4Jd5+msAmxn1c8vwuHlKTMwYXzEQmTcOjIa7Ad?=
 =?us-ascii?Q?ebHoQ9w9gkHDFKT6Fo4S8t95/u+gLXZROSjm/VjxLwQ5SFSsmR2fZqC1ReYB?=
 =?us-ascii?Q?VNNnL/lMlQPNfXtyZR5T2uuC9gtq/rbzNsAHjjBcYoksQ80aqGoA7tFddz61?=
 =?us-ascii?Q?gekx88Ub0jGfS6Psik7Vi4m9X34lN3KinYO1JhRXvAqvAjU5wBTD6y9JnZDU?=
 =?us-ascii?Q?entlie4izB03rHnNE2UpqpJb8o1GI1f+rlZCD5mpo+cLPX2PcOcIWjEz422l?=
 =?us-ascii?Q?S1tZpVFm2BJV/D7+oLkEWaZg5yD+2DlFybUZzfE9jBOjpe9AlXLSyKXRdOFF?=
 =?us-ascii?Q?cWJ9y5sD4LH2rJMELNDEe1tz/+SMGBvBt2fIUVIff2K/D39pX1EZ3YinGuRd?=
 =?us-ascii?Q?KLARml2DKaslZOfD2peOl8mqsBcYhqwlVm7ePpYMcnL+Jada2s3439C2H9ul?=
 =?us-ascii?Q?DYefGSycJuuAWlLsVEE9GgYGx2O5JFiXRI/QjXkU+BPc4ULG2Lugty4Q6YSy?=
 =?us-ascii?Q?OAfV24oGht50296xs7jVWkEvt2VN7wf7gR/uPiQP3dUkxnAnC5GEtorfr1AL?=
 =?us-ascii?Q?VqI4zimG9umj2hKxrAYQ/EqR1fNI7p3sT3FJO1hBP4zIlsNrHKHIMRhrR+2/?=
 =?us-ascii?Q?ekTydULm9DbsQ4YHDOEclR5S5zWFvqD86FN5KUvlNS7se86w2g+1GQZ1hPuA?=
 =?us-ascii?Q?aICoMl4se8k9OljgIQxP4OK+/Ky0ere4Sx79RE8ZGTxU4w0ePWsnQ5dX5IVz?=
 =?us-ascii?Q?P59u1lCLriWsDxlvdq8nM5L8Xigp2qbPCB/F6w62FyzT0nq81dvoVLhLTeky?=
 =?us-ascii?Q?RBjIQ2xdN3yuUqTXAxXwuZ8iTdNw1hysUtb5FxsO?=
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0d562459-f5de-48fa-d69a-08dc27f62422
X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB9642.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Feb 2024 16:02:10.3320
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JOESTTIhfRUxHZfgfPYVgRRBmGvOmRYh/V45IQBrnrXCqIEsRZ566acRgAg7288Cw4Ks2NyO6j1vv3PPZPiVPg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM7PR04MB7109

On Wed, Feb 07, 2024 at 03:37:30PM +0300, Serge Semin wrote:
> On Tue, Feb 06, 2024 at 05:47:26PM -0500, Frank Li wrote:
> > On Mon, Feb 05, 2024 at 02:13:37PM -0500, Frank Li wrote:
> > > On Mon, Feb 05, 2024 at 06:30:48PM +0000, Rob Herring wrote:
> > > > On Sat, Feb 03, 2024 at 01:44:31AM +0300, Serge Semin wrote:
> > > > > On Fri, Feb 02, 2024 at 10:11:27AM -0500, Frank Li wrote:
> > > > > > Add an outbound iATU-capable memory-region which will be used to send PCIe
> > > > > > message (such as PME_Turn_Off) to peripheral. So all platforms can use
> > > > > > common method to send out PME_Turn_Off message by using one outbound iATU.
> > > > > > 
> > > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > > > ---
> > > > > >  Documentation/devicetree/bindings/pci/snps,dw-pcie.yaml | 4 ++++
> > > > > >  1 file changed, 4 insertions(+)
> > > > > > 
> > > > > > diff --git a/Documentation/devicetree/bindings/pci/snps,dw-pcie.yaml b/Documentation/devicetree/bindings/pci/snps,dw-pcie.yaml
> > > > > > index 022055edbf9e6..25a5420a9ce1e 100644
> > > > > > --- a/Documentation/devicetree/bindings/pci/snps,dw-pcie.yaml
> > > > > > +++ b/Documentation/devicetree/bindings/pci/snps,dw-pcie.yaml
> > > > > > @@ -101,6 +101,10 @@ properties:
> > > > > 
> > > > > >              Outbound iATU-capable memory-region which will be used to access
> > > > > >              the peripheral PCIe devices configuration space.
> > > > > >            const: config
> > > > > > +        - description:
> > > > > > +            Outbound iATU-capable memory-region which will be used to send
> > > > > > +            PCIe message (such as PME_Turn_Off) to peripheral.
> > > > > > +          const: msg
> > > > > 
> > > > > Note there is a good chance Rob won't like this change. AFAIR he
> > > > > already expressed a concern regarding having the "config" reg-name
> > > > > describing a memory space within the outbound iATU memory which is
> > > > > normally defined by the "ranges" property. Adding a new reg-entry with
> > > > > similar semantics I guess won't receive warm welcome.
> > > > 
> > > > I do think it is a bit questionable. Ideally, the driver could 
> > > > just configure this on its own. However, since we don't describe all of 
> > > > the CPU address space (that's input to the iATU) already, that's not 
> > > > going to be possible. I suppose we could fix that, but then config space 
> > > > would have to be handled differently too.
> > > 
> > > Sorry, I have not understand what your means. Do you means, you want
> > > a "cpu-space", for example, 0x8000000 - 0x9000000 for all ATU. 
> > > 
> > > Then allocated some space to 'config', 'io', 'memory' and this 'msg'.
> > 
> > @rob:
> > 
> >     So far, I think "msg" is feasilbe solution. Or give me some little
> > detail direction?
> 
> Found the Rob' note about the iATU-space chunks utilized in the reg
> property:
> https://lore.kernel.org/linux-pci/CAL_JsqLp7QVgxrAZkW=z38iB7SV5VeWH1O6s+DVCm9p338Czdw@mail.gmail.com/
> 
> So basically Rob meant back then that
> either originally we should have defined a new reg-name like "atu-out"
> with the entire outbound iATU CPU-space specified and unpin the
> regions like "config"/"ecam"/"msg"/etc from there in the driver
> or, well, stick to the chunking further. The later path was chosen
> after the patch with the "ecam" reg-name was accepted (see the link
> above).
> 
> Really ECAM/config space access, custom TLP messages, legacy interrupt
> TLPs, etc are all application-specific features. Each of them is
> implemented based on a bit specific but basically the same outbound
> iATU engine setup. Thus from the "DT is a hardware description" point
> of view it would have been enough to describe the entire outbound iATU
> CPU address space and then let the software do the space
> reconfiguration in runtime based on it' application needs.

There are "addr_space" in EP mode, which useful map out outbound iatu
region. We can reuse this name.

To keep compatiblity, cut hole from 'config' and 'ranges'. If there are
not 'config', we can alloc a 1M(default) from top for 'config', then, 4K
(default) for msg, 64K( for IO if not IO region in 'ranges'), left is
mem region. We can config each region size by module parameter or drvdata.

So we can deprecate 'config', even 'ranges'

> 
> * Rob, correct me if am wrong.
> 
> On the other hand it's possible to have more than one disjoint CPU
> address region handled by the outbound iATU (especially if there is no
> AXI-bridge enabled, see XALI - application transmit client interfaces
> in HW manual). Thus having a single reg-property might get to be
> inapplicable in some cases. Thinking about that got me to an idea.
> What about just extending the PCIe "ranges" property flags
> (IORESOURCE_TYPE_BITS) with the new ones in this case indicating the
> TLP Msg mapping? Thus we can avoid creating app-specific reg-names and
> use the flag to define a custom memory range for the TLP messages
> generation. At some point it can be also utilized for the config-space
> mapping. What do you think?

IORESOURCE_TYPE_BITS is 1f, Only 5bit. If extend IORESOURCE_TYPE_BITS, 
all IORESOURCE_* bit need move. And it is actual MEMORY regain. 

Or we can use IORESOURCE_BITS (0xff)

/* PCI ROM control bits (IORESOURCE_BITS) */
#define IORESOURCE_ROM_ENABLE		(1<<0)	/* ROM is enabled, same as PCI_ROM_ADDRESS_ENABLE */
#define IORESOURCE_ROM_SHADOW		(1<<1)	/* Use RAM image, not ROM BAR */

/* PCI control bits.  Shares IORESOURCE_BITS with above PCI ROM.  */
#define IORESOURCE_PCI_FIXED		(1<<4)	/* Do not move resource */
#define IORESOURCE_PCI_EA_BEI		(1<<5)	/* BAR Equivalent Indicator */

we can add

IORESOURCE_PRIV_WINDOWS			(1<<6)

I think previous method was more extendable. How do you think?

> 
> -Serge(y)
> 
> > 
> > Frank
> > 
> > > 
> > > Frank
> > > 
> > > > 
> > > > Rob