Received-SPF: pass (google.com: domain of linux-kernel+bounces-154417-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Mon, 22 Apr 2024 19:20:48 -0700
From: Dan Williams <dan.j.williams@intel.com>
To: Robert Richter <rrichter@amd.com>, "Rafael J. Wysocki"
	<rafael@kernel.org>, Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar
	<mingo@redhat.com>, Borislav Petkov <bp@alien8.de>, Dave Hansen
	<dave.hansen@linux.intel.com>, <x86@kernel.org>, Andy Lutomirski
	<luto@kernel.org>, Peter Zijlstra <peterz@infradead.org>, Dan Williams
	<dan.j.williams@intel.com>, Alison Schofield <alison.schofield@intel.com>
CC: <linux-acpi@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
	<linux-cxl@vger.kernel.org>, Robert Richter <rrichter@amd.com>, Derick Marks
	<derick.w.marks@intel.com>, "H. Peter Anvin" <hpa@zytor.com>
Subject: RE: [PATCH v3 1/5] x86/numa: Fix SRAT lookup of CFMWS ranges with
 numa_fill_memblks()
Message-ID: <66271b0072317_69102944c@dwillia2-xfh.jf.intel.com.notmuch>
References: <20240419140203.1996635-1-rrichter@amd.com>
 <20240419140203.1996635-2-rrichter@amd.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240419140203.1996635-2-rrichter@amd.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?CLjTvtUnXiP/QFfMqeQAdu3AmRp6PoEztESfZWiD3FK64y58ZcKNHDJuJmnI?=
 =?us-ascii?Q?og3QL12rolIZK2dualqh9nQFjD7pfYNJ1YPmDxT7A4pcNMq2wdcxVcFjeZCU?=
 =?us-ascii?Q?FAysK1Y34bdupp27TO/sDVHsjI7sXOQTzjc5gCIrBLPYpUhbX+ImeW4VJHWQ?=
 =?us-ascii?Q?avCb0//wQWFCtZbt9ZtQp6krT3qXnt2t49vrZyKXSztwXRB/x17evEW8Q+13?=
 =?us-ascii?Q?LhIUVB9Ug16FmcfGnJA3Lu9I3WgSZEUdgWPcPx16VrSWMqv69BclOzSI/FgR?=
 =?us-ascii?Q?P0zWPmCGGbhAZzPTX8YYiMvDFxYC0lwXcDM0U3D20ZDTv4n7ybdz+m0eVqys?=
 =?us-ascii?Q?EL7uQ+gvHCmaU58o+ni6rQdkm8bIrL5qdfdsoGiShACWilQhrGF2TJzA8l1f?=
 =?us-ascii?Q?+NJf01Z57yvrC9cLG5jmgLWE3OCndmmlzkHHrr0HsWJxs3D9XvMAbIPvB0P1?=
 =?us-ascii?Q?pBMdkU2Htf7F0Fhyme6iM2bzdExDy7R0eFlJFF1RMRmMdSpqfppwmVee4RkS?=
 =?us-ascii?Q?bpu6ULSMQ1qlYWGJ9lwdGxryKA8d8VXyzQZ+Pz5SuJuIdwSijoADstmU3pXE?=
 =?us-ascii?Q?RgyDpzy4uVspeOMHfLQt0k86EGvOww07VT2HEt+K3HcIlUqUpsmmTGDhRhUx?=
 =?us-ascii?Q?LOg5SD/68XpmrFG2RidweQEhQLw9L5Kk1Q6XVxziqV+UmMXO5hqot754GgS+?=
 =?us-ascii?Q?8H+w7mDDhjUwMzEL4c2ShRzNBGXC83NHMUH0BI/KPMxm+/+8hMMYHH+tljBq?=
 =?us-ascii?Q?qI3qstrKRFCy47GRWzwkcd8DjgWCkamfVpC0Z57ldKUhIq0jzvFBGl8goqI+?=
 =?us-ascii?Q?nUB6JTeTCJripm9rJeZgPtWJ8AmYEQ5WtWGPtXNYqt3BjUfD5X8OcogMHS2F?=
 =?us-ascii?Q?ANO3zfh6lG0eZGCHGjTqhl0e4hCAEM5ezeCqDke5GkVLKs73om45VG/VEY6Z?=
 =?us-ascii?Q?aHFU9Kl7Hm/oJ3Y6/Z8DfXhXi53e5pVvGElcXTVPD3SzcVAGTa4Zsz/yF6hX?=
 =?us-ascii?Q?aWHvw16+BfkFOVyB/8cuJH5EwBw3f0UegaOc9r2AnYc7augkUz1pJTn8dz7C?=
 =?us-ascii?Q?GPm+spC5pjoTeszwh4bpnoOJEcTwmfjhdlx3vE5ElOhgMK0ut7VIKeVwFUen?=
 =?us-ascii?Q?1MZ5A+qpK1pvpb66XwQyeYBxbPitixVooeZWU0lUyfCchR7pYB8iCm6QSIfj?=
 =?us-ascii?Q?T1+NmzEnbcNyYuvKR8TElgGPZTcGxvkBDRnIoOMGErKkJysP/i6OA14r2/qN?=
 =?us-ascii?Q?ilM531BzjwFIyrjWEmJaxveyfraNGV5yLxIUc8pIVsyymBsoT3R1rklllboD?=
 =?us-ascii?Q?wMkAU1QColAENk4OZtV7Sx8lHzkrQgti4Nw/TLo56VYbkpipIOgIrmErvCtq?=
 =?us-ascii?Q?tgfV0VsFF684jwngSEPDcn9ohBFSTuq04TAtk0K6D9epARXyKjMqV3UGIpwm?=
 =?us-ascii?Q?VMaUU15RzKvhC5M3bnESrlmwCJQoX5hQLwucTkuXDOUNzhW9oO4geNvjZsVl?=
 =?us-ascii?Q?uwXkV2tf7wR/i76QJMCfI98iJlfjbI+U/pL1cGs0R8FXXxaCfLZ+cf/PO3+4?=
 =?us-ascii?Q?TVRzbcVzKHczh0ar22ziZ9bysdGp5gSeH1Z5bgsaSj8auR5NZJ5/jy5kZDhI?=
 =?us-ascii?Q?BQ=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 896390fe-0399-4146-ccb7-08dc633bfed1
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Apr 2024 02:20:51.0527
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: k+L4MbkuYkf1NLpMDINCVXpaglaTqBHkwchogjuHEeHpYcNCVBrpmH32LgpZBVfmh3LqwnsPnnHpGzyykHBZVfTZm7hTwiVpkTuaTFrd8RU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB4807
X-OriginatorOrg: intel.com

Robert Richter wrote:
> For configurations that have the kconfig option NUMA_KEEP_MEMINFO
> disabled, the SRAT lookup done with numa_fill_memblks() fails
> returning NUMA_NO_MEMBLK (-1). An existing SRAT memory range cannot be
> found for a CFMWS address range. This causes the addition of a
> duplicate numa_memblk with a different node id and a subsequent page
> fault and kernel crash during boot.
> 
> numa_fill_memblks() is implemented and used in the init section only.
> The option NUMA_KEEP_MEMINFO is only for the case when NUMA data will
> be used outside of init. So fix the SRAT lookup by moving
> numa_fill_memblks() out of the NUMA_KEEP_MEMINFO block to make it
> always available in the init section.
> 
> Note that the issue was initially introduced with [1]. But since
> phys_to_target_node() was originally used that returned the valid node
> 0, an additional numa_memblk was not added. Though, the node id was
> wrong too.

Wrong node seems worth notifying and we can figure out on the backend
where the backport makes sense.

> 
> [1] commit fd49f99c1809 ("ACPI: NUMA: Add a node and memblk for each
>     CFMWS not in SRAT")
> 
> Fixes: 8f1004679987 ("ACPI/NUMA: Apply SRAT proximity domain to entire CFMWS window")
> Cc: Derick Marks <derick.w.marks@intel.com>
> Cc: Dan Williams <dan.j.williams@intel.com>
> Cc: Alison Schofield <alison.schofield@intel.com>
> Signed-off-by: Robert Richter <rrichter@amd.com>
> ---
>  arch/x86/include/asm/sparsemem.h | 2 +-
>  arch/x86/mm/numa.c               | 4 ++--
>  2 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/x86/include/asm/sparsemem.h b/arch/x86/include/asm/sparsemem.h
> index 1be13b2dfe8b..1aaa447ef24b 100644
> --- a/arch/x86/include/asm/sparsemem.h
> +++ b/arch/x86/include/asm/sparsemem.h
> @@ -37,9 +37,9 @@ extern int phys_to_target_node(phys_addr_t start);
>  #define phys_to_target_node phys_to_target_node
>  extern int memory_add_physaddr_to_nid(u64 start);
>  #define memory_add_physaddr_to_nid memory_add_physaddr_to_nid
> +#endif
>  extern int numa_fill_memblks(u64 start, u64 end);
>  #define numa_fill_memblks numa_fill_memblks

It just feels like numa_fill_memblks() has absolutely no business being
defined in arch/x86/include/asm/sparsemem.h.

The only use for numa_fill_memblks() is to arrange for NUMA nodes to be
applied to memory ranges hot-onlined by the CXL driver.

It belongs right next to numa_add_memblk(), and I suspect
arch/x86/include/asm/sparsemem.h was only chosen to avoid figuring out
what to do about the fact that linux/numa.h does not include asm/numa.h
and that all implementations either provide numa_add_memblk() or select
the generic implementation.

So I would prefer that this do the proper fix and get
numa_fill_memblks() completely out of the sparsemem.h path.

Something like the following which boots for me:

diff --git a/arch/x86/include/asm/numa.h b/arch/x86/include/asm/numa.h
index ef2844d69173..12a93a3466c4 100644
--- a/arch/x86/include/asm/numa.h
+++ b/arch/x86/include/asm/numa.h
@@ -26,6 +26,7 @@ extern s16 __apicid_to_node[MAX_LOCAL_APIC];
 extern nodemask_t numa_nodes_parsed __initdata;
 
 extern int __init numa_add_memblk(int nodeid, u64 start, u64 end);
+extern int __init numa_fill_memblks(u64 start, u64 end);
 extern void __init numa_set_distance(int from, int to, int distance);
 
 static inline void set_apicid_to_node(int apicid, s16 node)
diff --git a/arch/x86/include/asm/sparsemem.h b/arch/x86/include/asm/sparsemem.h
index 1be13b2dfe8b..64df897c0ee3 100644
--- a/arch/x86/include/asm/sparsemem.h
+++ b/arch/x86/include/asm/sparsemem.h
@@ -37,8 +37,6 @@ extern int phys_to_target_node(phys_addr_t start);
 #define phys_to_target_node phys_to_target_node
 extern int memory_add_physaddr_to_nid(u64 start);
 #define memory_add_physaddr_to_nid memory_add_physaddr_to_nid
-extern int numa_fill_memblks(u64 start, u64 end);
-#define numa_fill_memblks numa_fill_memblks
 #endif
 #endif /* __ASSEMBLY__ */
 
diff --git a/drivers/acpi/numa/srat.c b/drivers/acpi/numa/srat.c
index e45e64993c50..3b09fd39eeb4 100644
--- a/drivers/acpi/numa/srat.c
+++ b/drivers/acpi/numa/srat.c
@@ -208,6 +208,11 @@ int __init srat_disabled(void)
 	return acpi_numa < 0;
 }
 
+__weak int __init numa_fill_memblks(u64 start, u64 end)
+{
+	return NUMA_NO_MEMBLK;
+}
+
 #if defined(CONFIG_X86) || defined(CONFIG_ARM64) || defined(CONFIG_LOONGARCH)
 /*
  * Callback for SLIT parsing.  pxm_to_node() returns NUMA_NO_NODE for
diff --git a/include/linux/numa.h b/include/linux/numa.h
index 915033a75731..8485d98e554d 100644
--- a/include/linux/numa.h
+++ b/include/linux/numa.h
@@ -36,13 +36,6 @@ int memory_add_physaddr_to_nid(u64 start);
 int phys_to_target_node(u64 start);
 #endif
 
-#ifndef numa_fill_memblks
-static inline int __init numa_fill_memblks(u64 start, u64 end)
-{
-	return NUMA_NO_MEMBLK;
-}
-#endif
-
 #else /* !CONFIG_NUMA */
 static inline int numa_nearest_node(int node, unsigned int state)
 {