Received-SPF: pass (google.com: domain of linux-kernel+bounces-162826-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
From: Josua Mayer <josua@solid-run.com>
Date: Mon, 29 Apr 2024 20:24:36 +0200
Subject: [PATCH RFC v2] arm64: dts: lx2160a: extend 32-bit, and add 64-bit
 pci regions
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
Message-Id: <20240429-lx2160-pci-v2-1-1b94576d6263@solid-run.com>
To: Shawn Guo <shawnguo@kernel.org>, Li Yang <leoyang.li@nxp.com>, 
 Rob Herring <robh+dt@kernel.org>, 
 Krzysztof Kozlowski <krzysztof.kozlowski+dt@linaro.org>, 
 Conor Dooley <conor+dt@kernel.org>
Cc: Yazan Shhady <yazan.shhady@solid-run.com>, 
 Jon Nettleton <jon@solid-run.com>, linux-arm-kernel@lists.infradead.org, 
 devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, 
 Josua Mayer <josua@solid-run.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?QmZzOXhPdGZZM1lNWkROMWRiSW55Q1d4diszNXFhTjVYY3FjY0dNZklYNFlD?=
 =?utf-8?B?ZzZmM1pESnZBRU1jY2FWRVRlVkNxYmVLM2RHb2Q3MzRtSTk3YXl1Y25lWUN4?=
 =?utf-8?B?Lyt5VGVGZitVVUI3dXBVcWJFQVYveVpyaHQwbS9jQ3NGU1F3ZlgrT2ZSRHpH?=
 =?utf-8?B?N0JDVWdYWXFJdXNDckF4OWRaeDNjYnpNaXQ2SnN2Z05tVWo4UUJJSTFsWkNk?=
 =?utf-8?B?aEg4NXJqMHNSczhZZnd0aW9aVFdHNHZQV1hKbGcxeUtiVnR3Nmk5RnJOanZV?=
 =?utf-8?B?VFFHT3dMTjFEZEF5V1RUUjlVai9yWHNsTkVXejFqZGYvRVlqWklVa1JVVVd1?=
 =?utf-8?B?MkxFT2ZZa2luK0xqQXZvRFdLZmw2bytSSmlzMTAyaHZ4N1ZPNGNhSXlYaDhO?=
 =?utf-8?B?aTdNTmsyOVdpTzJmVU9JRk02YUpNaVBZdVVXOWFRbENibUpkbVpMSXkrTXJi?=
 =?utf-8?B?UUdIdjl1ODNkRDBrR3NJSGpzRitaSWJSZ21rWEdmOVF1VzIxcHNZZjlqV3lp?=
 =?utf-8?B?ZlBhMVNyOC9ZLzhuaEJ2cy9sU0M4VjVGYmdEbE05ZUZjSnZxZDdmRUU1YlZO?=
 =?utf-8?B?WGE1ampabWE3K0RwU2pwSWZjbkt3ajc5Q3Q0dlVFalcxUGt4Q0k4SjJtTzho?=
 =?utf-8?B?WUlFWEhoWjgveGk5V2h6U25XRVd2VVFudkVGelRvKzRlMXY0RHpFb3FKdzFu?=
 =?utf-8?B?ZWJ4azdJVmxGTDFQYmNsczVGSjlpamlKZnhRZUo1OXZsTG5lNTh2elFtNU9T?=
 =?utf-8?B?Wkp5dFNLdzhPZUpEQ2VoL3orendRRWZQTkx3Z2F1d0tkRHAwNEhFR2pXbG5Y?=
 =?utf-8?B?K0FQaFZvL09wM1RaYU1MVjRUVUZkQkFrMkJlTi9saG5rUmtaT29nMFRwRTRm?=
 =?utf-8?B?Q0grMzVzU2VPVTJqY3Uyb2ZCenpRTGU5UXhIWmZwT0hlOHBTT0pmZUtkSXM4?=
 =?utf-8?B?RDMrU0RWUlN4dE5HeXlIcU9YVzJwUzZxclZaakM2b3lhK3NOVFB6OTNXYWtH?=
 =?utf-8?B?RGZET2lMbWRlb2k1L25oMElzOHFvdWt4bG9aT1hSc3BGNEVnVkt2ZFNENTFk?=
 =?utf-8?B?RXIrekpLdi9XeVQ3RnNnRXlWQW5aNU5PdVZyR2pLQ05KSFRsaVJESXlzNXhi?=
 =?utf-8?B?ZHluU2pZQ1gyb09LQTJYOExYMmpHV1d3R2o0YkJFVUJOTFEzVW1xUkpnTUtw?=
 =?utf-8?B?Y25BOXNkNmdhbWptNGhzZ212M0g5eHRKWnRaN1ZWS09paHVsdFN2dFNIQzJC?=
 =?utf-8?B?M3NMYXFhZmJnT1V0bE1PZnFNazA5aGhJWHkwaXlzY2dYQkNTbThYd2M2N0VJ?=
 =?utf-8?B?TU5aTVNZemJmbGNYckxHenRqVmgxdTk2cnB5N0dKSzZaVW1VR3dhU3YxZEtt?=
 =?utf-8?B?anNlZytuZWR2UVhLeUhFSUZDVU9UcmZaY3R5VW5mMWtGL3dTcnVVZnhURk5p?=
 =?utf-8?B?Q1BDL3BKMit2SE85QWdiWmpTM3U5cVhtd0NmSFZQdm83Sm5ZMTRYSHQ2THBY?=
 =?utf-8?B?aXQ4Y0hJL0lqYTQ2ZVRFbGYzU1RHaTNxeWs0VUkxOXZYZ3FmakZjelI1NDR1?=
 =?utf-8?B?TjBPOVhCMFBudWhtTU5FRTVPRmNVbUFSa2hKRXdvbFhlc3paUGIxRlZzNjVR?=
 =?utf-8?B?UHJoU1VnRVk1dy9IZi92WXNGRlZlV3ZSMk1uZitGQTZTaHRiNlYxRnBqcS9B?=
 =?utf-8?B?WEM0SFlWSGdOYU5vUTc2aENPdGgvbHB3YXU4bjB2UjZwMXFlY3RDUHl1TmMw?=
 =?utf-8?B?NzQzaHlnZzB5OGpoT1ByT2MvNHI3RUJNK2hpNllXOHhqaHJ2dkxvWHI4NWhY?=
 =?utf-8?B?OUYzOGlQbTA0ZGx0THRsdnVHRkl6TEZ5SVRzWEZ4Z09EamVlbTJxdmkzcVJ0?=
 =?utf-8?B?WmYxYmJvUDV4cElLWDJuckE3T1NwV0FBbmJUbm5teWhKR3ZVRHhzSzYvQ1dw?=
 =?utf-8?B?UXc1VWs0L0E1b3F3WDAwSTlha0dhcFBvWUxCVVNYdlFJNE1sR3NlejZXeGg2?=
 =?utf-8?B?QTJNRUlodjExbnhDR2FJQ2R6QVQycFQzOHloQkM5Y2p5TnlJR0VNdDRYZ1Q4?=
 =?utf-8?B?N2ZleWVqODM0bUZSQ2pIcXB2YVVtWFRSMDRETDBQSFRURFRhT2NkU21PNng3?=
 =?utf-8?Q?9zjgIg1ECBLlfYwEDFcHh3adw?=
X-OriginatorOrg: solid-run.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 975be4b2-135e-4d88-06d1-08dc6879a424
X-MS-Exchange-CrossTenant-AuthSource: AM9PR04MB7586.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Apr 2024 18:24:43.4782
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: a4a8aaf3-fd27-4e27-add2-604707ce5b82
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: nZjEmky46SOgGFSqUMVLApkm4wsKB3Wf9U2LPelc+LJBL8Li75vdqN8n+i3WG+Uf1mJ6UBVQ5e3ekL6rGe+fPg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR04MB7782

LX2160 SoC pci-e controller supports 64-bit memory regions up to 16GB,
32-bit regions up to 3GB and 16-bit regions up to 64k.

For each pci-e controller:
- extend the existing 32-bit regions to 3GB size
- add 16-bit region
- add 64-bit region

Same memory allocation with similar flags were been tested with UEFI
and ACPI on pcie3 and pcie5.
This specific device-tree configuration was tested with nxp lsdk-21.08
based u-boot:
- pcie5 with a Radeon Pro WX2100 with Gnome Desktop
- pcie3 with an ADATA NVME

Fixes allocation of large, and 64-bit BARs as requested by many pci
cards, especially graphics processors or AI accelerators, e.g.:
[    2.941187] pci 0000:01:00.0: BAR 0: no space for [mem size 0x200000000 64bit pref]
[    2.948834] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x200000000 64bit pref]

This version is still marked RFC because as it carries a workaround for
a limitation of the designware pcie controller driver.
The ATU has a maximum allocation size of 4GB. Larger allocations should
be implemented as multiple allocations of 4GB in the driver,
similar to how UEFI implemented it for ACPI.

Signed-off-by: Josua Mayer <josua@solid-run.com>
---
Changes in v2:
- adjusted flags to fix several errors during probe and bar allocation
- explicitly tested with 2 pci cards on Debian (Linux 6.1)
- still rfc because a limitation in designware pci driver
- Link to v1: https://lore.kernel.org/r/20240321-lx2160-pci-v1-1-3673708f7eb6@solid-run.com
---
 arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi | 55 +++++++++++++++++++++++---
 1 file changed, 49 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi b/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
index 6640b49670ae..ec4e6252f83b 100644
--- a/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
+++ b/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
@@ -1,3 +1,4 @@
+
 // SPDX-License-Identifier: (GPL-2.0 OR MIT)
 //
 // Device Tree Include file for Layerscape-LX2160A family SoC.
@@ -1134,7 +1135,14 @@ pcie1: pcie@3400000 {
 			apio-wins = <8>;
 			ppio-wins = <8>;
 			bus-range = <0x0 0xff>;
-			ranges = <0x82000000 0x0 0x40000000 0x80 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+			// ranges = <0x02102000 0x84 0x00000000 0x84 0x00000000 0x04 0x00000000>, /* 64-Bit - prefetchable */
+			/* split 64-bit area into 4GB chunks as workaround for ATU max allocation size */
+			ranges = <0x02102000 0x87 0x00000000 0x87 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0x86 0x00000000 0x86 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0x85 0x00000000 0x85 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0x84 0x00000000 0x84 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02000200 0x00 0x40000000 0x80 0x40000000 0x00 0xc0000000>, /* 32-Bit - non-prefetchable */
+				 <0x01200100 0x00 0x00000000 0x80 0x10000000 0x00 0x00010000>; /* 16-Bit IO Window */
 			msi-parent = <&its>;
 			#interrupt-cells = <1>;
 			interrupt-map-mask = <0 0 0 7>;
@@ -1162,7 +1170,14 @@ pcie2: pcie@3500000 {
 			apio-wins = <8>;
 			ppio-wins = <8>;
 			bus-range = <0x0 0xff>;
-			ranges = <0x82000000 0x0 0x40000000 0x88 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+			// ranges = <0x02102000 0x8c 0x00000000 0x8c 0x00000000 0x04 0x00000000>, /* 64-Bit - prefetchable */
+			/* split 64-bit area into 4GB chunks as workaround for ATU max allocation size */
+			ranges = <0x02102000 0x8f 0x00000000 0x8f 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0x8e 0x00000000 0x8e 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0x8d 0x00000000 0x8d 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0x8c 0x00000000 0x8c 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02000200 0x00 0x40000000 0x88 0x40000000 0x00 0xc0000000>, /* 32-Bit - non-prefetchable */
+				 <0x01200100 0x00 0x00000000 0x88 0x10000000 0x00 0x00010000>; /* 16-Bit IO Window */
 			msi-parent = <&its>;
 			#interrupt-cells = <1>;
 			interrupt-map-mask = <0 0 0 7>;
@@ -1190,7 +1205,14 @@ pcie3: pcie@3600000 {
 			apio-wins = <256>;
 			ppio-wins = <24>;
 			bus-range = <0x0 0xff>;
-			ranges = <0x82000000 0x0 0x40000000 0x90 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+			// ranges = <0x02102000 0x94 0x00000000 0x94 0x00000000 0x04 0x00000000>, /* 64-Bit - prefetchable */
+			/* split 64-bit area into 4GB chunks as workaround for ATU max allocation size */
+			ranges = <0x02102000 0x97 0x00000000 0x97 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0x96 0x00000000 0x96 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0x95 0x00000000 0x95 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0x94 0x00000000 0x94 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02000200 0x00 0x40000000 0x90 0x40000000 0x00 0xc0000000>, /* 32-Bit - non-prefetchable */
+				 <0x01200100 0x00 0x00000000 0x90 0x10000000 0x00 0x00010000>; /* 16-Bit IO Window */
 			msi-parent = <&its>;
 			#interrupt-cells = <1>;
 			interrupt-map-mask = <0 0 0 7>;
@@ -1218,7 +1240,14 @@ pcie4: pcie@3700000 {
 			apio-wins = <8>;
 			ppio-wins = <8>;
 			bus-range = <0x0 0xff>;
-			ranges = <0x82000000 0x0 0x40000000 0x98 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+			// ranges = <0x02102000 0x9c 0x00000000 0x9c 0x00000000 0x04 0x00000000>, /* 64-Bit - prefetchable */
+			/* split 64-bit area into 4GB chunks as workaround for ATU max allocation size */
+			ranges = <0x02102000 0x9f 0x00000000 0x9f 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0x9e 0x00000000 0x9e 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0x9d 0x00000000 0x9d 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0x9c 0x00000000 0x9c 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02000200 0x00 0x40000000 0x98 0x40000000 0x00 0xc0000000>, /* 32-Bit - non-prefetchable */
+				 <0x01200100 0x00 0x00000000 0x98 0x10000000 0x00 0x00010000>; /* 16-Bit IO Window */
 			msi-parent = <&its>;
 			#interrupt-cells = <1>;
 			interrupt-map-mask = <0 0 0 7>;
@@ -1246,7 +1275,14 @@ pcie5: pcie@3800000 {
 			apio-wins = <256>;
 			ppio-wins = <24>;
 			bus-range = <0x0 0xff>;
-			ranges = <0x82000000 0x0 0x40000000 0xa0 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+			// ranges = <0x02102000 0xa4 0x00000000 0xa4 0x00000000 0x04 0x00000000>, /* 64-Bit - prefetchable */
+			/* split 64-bit area into 4GB chunks as workaround for ATU max allocation size */
+			ranges = <0x02102000 0xa7 0x00000000 0xa7 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0xa6 0x00000000 0xa6 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0xa5 0x00000000 0xa5 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0xa4 0x00000000 0xa4 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02000200 0x00 0x40000000 0xa0 0x40000000 0x00 0xc0000000>, /* 32-Bit - non-prefetchable */
+				 <0x01200100 0x00 0x00000000 0xa0 0x10000000 0x00 0x00010000>; /* 16-Bit IO Window */
 			msi-parent = <&its>;
 			#interrupt-cells = <1>;
 			interrupt-map-mask = <0 0 0 7>;
@@ -1274,7 +1310,14 @@ pcie6: pcie@3900000 {
 			apio-wins = <8>;
 			ppio-wins = <8>;
 			bus-range = <0x0 0xff>;
-			ranges = <0x82000000 0x0 0x40000000 0xa8 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+			// ranges = <0x02102000 0xac 0x00000000 0xac 0x00000000 0x04 0x00000000>, /* 64-Bit - prefetchable */
+			/* split 64-bit area into 4GB chunks as workaround for ATU max allocation size */
+			ranges = <0x02102000 0xaf 0x00000000 0xaf 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0xae 0x00000000 0xae 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0xad 0x00000000 0xad 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02102000 0xac 0x00000000 0xac 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+				 <0x02000200 0x00 0x40000000 0xa8 0x40000000 0x00 0xc0000000>, /* 32-Bit - non-prefetchable */
+				 <0x01200100 0x00 0x00000000 0xa8 0x10000000 0x00 0x00010000>; /* 16-Bit IO Window */
 			msi-parent = <&its>;
 			#interrupt-cells = <1>;
 			interrupt-map-mask = <0 0 0 7>;

---
base-commit: e8f897f4afef0031fe618a8e94127a0934896aba
change-id: 20240118-lx2160-pci-4bdb196e58f3

Sincerely,
-- 
Josua Mayer <josua@solid-run.com>