From: Alison Schofield <[email protected]>
Changes in v4:
- Remove useless export of numa_fill_memblks() (Dan)
- Rebase on latest tip tree
v3: https://lore.kernel.org/linux-cxl/[email protected]/
----
Cover Letter:
The CXL subsystem requires the creation of NUMA nodes for CFMWS
Windows[1] not described in the SRAT. The existing implementation
only addresses windows that the SRAT describes completely or not
at all. This work addresses the case of partially described CFMWS
Windows by extending proximity domains in a portion of a CFMWS
window to the entire window.
Introduce a NUMA helper, numa_fill_memblks(), to fill gaps in a
numa_meminfo memblk address range. Update the CFMWS parsing in the
ACPI driver to use numa_fill_memblks() to extend SRAT defined
proximity domains to entire CXL windows.
An RFC of this patchset was previously posted for CXL folks review
here[2]. The RFC feedback led to the implementation here, extending
existing memblks (Dan). Also, both Jonathan and Dan influenced the
changelog comments in the ACPI patch, with regards to setting
expectations on this evolving heuristic.
Repeating here to set reviewer expectations:
*Note that this heuristic will evolve when CFMWS Windows present a
wider range of characteristics. The extension of the proximity domain,
implemented here, is likely a step in developing a more sophisticated
performance profile in the future.
[1] CFMWS is defined in CXL Spec 3.0 Section 9.17.1.3 :
https://www.computeexpresslink.org/spec-landing
A CXL Fixed Memory Window is a region of Host Physical Address (HPA)
Space which routes accesses to CXL Host bridges. The 'S', of CFMWS,
stand for the structure that describes the window, hence it's common
name, CFMWS.
[2] https://lore.kernel.org/linux-cxl/[email protected]/
Alison Schofield (2):
x86/numa: Introduce numa_fill_memblks()
ACPI: NUMA: Apply SRAT proximity domain to entire CFMWS window
arch/x86/include/asm/sparsemem.h | 2 +
arch/x86/mm/numa.c | 80 ++++++++++++++++++++++++++++++++
drivers/acpi/numa/srat.c | 11 +++--
include/linux/numa.h | 7 +++
4 files changed, 97 insertions(+), 3 deletions(-)
base-commit: ac442f6a364dd23bc08086f07b4bc4ef8476a9fe
--
2.37.3
From: Alison Schofield <[email protected]>
Commit fd49f99c1809 ("ACPI: NUMA: Add a node and memblk for each
CFMWS not in SRAT") did not account for the case where the BIOS
only partially describes a CFMWS Window in the SRAT. That means
the omitted address ranges, of a partially described CFMWS Window,
do not get assigned to a NUMA node.
Replace the call to phys_to_target_node() with numa_add_memblks().
Numa_add_memblks() searches an HPA range for existing memblk(s)
and extends those memblk(s) to fill the entire CFMWS Window.
Extending the existing memblks is a simple strategy that reuses
SRAT defined proximity domains from part of a window to fill out
the entire window, based on the knowledge* that all of a CFMWS
window is of a similar performance class.
*Note that this heuristic will evolve when CFMWS Windows present
a wider range of characteristics. The extension of the proximity
domain, implemented here, is likely a step in developing a more
sophisticated performance profile in the future.
There is no change in behavior when the SRAT does not describe
the CFMWS Window at all. In that case, a new NUMA node with a
single memblk covering the entire CFMWS Window is created.
Fixes: fd49f99c1809 ("ACPI: NUMA: Add a node and memblk for each CFMWS not in SRAT")
Reported-by: Derick Marks <[email protected]>
Suggested-by: Dan Williams <[email protected]>
Signed-off-by: Alison Schofield <[email protected]>
Tested-by: Derick Marks <[email protected]>
---
drivers/acpi/numa/srat.c | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
diff --git a/drivers/acpi/numa/srat.c b/drivers/acpi/numa/srat.c
index 1f4fc5f8a819..12f330b0eac0 100644
--- a/drivers/acpi/numa/srat.c
+++ b/drivers/acpi/numa/srat.c
@@ -310,11 +310,16 @@ static int __init acpi_parse_cfmws(union acpi_subtable_headers *header,
start = cfmws->base_hpa;
end = cfmws->base_hpa + cfmws->window_size;
- /* Skip if the SRAT already described the NUMA details for this HPA */
- node = phys_to_target_node(start);
- if (node != NUMA_NO_NODE)
+ /*
+ * The SRAT may have already described NUMA details for all,
+ * or a portion of, this CFMWS HPA range. Extend the memblks
+ * found for any portion of the window to cover the entire
+ * window.
+ */
+ if (!numa_fill_memblks(start, end))
return 0;
+ /* No SRAT description. Create a new node. */
node = acpi_map_pxm_to_node(*fake_pxm);
if (node == NUMA_NO_NODE) {
--
2.37.3
alison.schofield@ wrote:
> From: Alison Schofield <[email protected]>
>
>
> Changes in v4:
> - Remove useless export of numa_fill_memblks() (Dan)
> - Rebase on latest tip tree
This thread has gone quiet. Any concerns from x86/mm folks if I take
this through the CXL tree with an x86 ack? Or anything else I can help
out with on this one?
>
> v3: https://lore.kernel.org/linux-cxl/[email protected]/
>
> ----
>
> Cover Letter:
>
> The CXL subsystem requires the creation of NUMA nodes for CFMWS
> Windows[1] not described in the SRAT. The existing implementation
> only addresses windows that the SRAT describes completely or not
> at all. This work addresses the case of partially described CFMWS
> Windows by extending proximity domains in a portion of a CFMWS
> window to the entire window.
[..]
alison.schofield@ wrote:
> From: Alison Schofield <[email protected]>
>
> Commit fd49f99c1809 ("ACPI: NUMA: Add a node and memblk for each
> CFMWS not in SRAT") did not account for the case where the BIOS
> only partially describes a CFMWS Window in the SRAT. That means
> the omitted address ranges, of a partially described CFMWS Window,
> do not get assigned to a NUMA node.
>
> Replace the call to phys_to_target_node() with numa_add_memblks().
> Numa_add_memblks() searches an HPA range for existing memblk(s)
> and extends those memblk(s) to fill the entire CFMWS Window.
>
> Extending the existing memblks is a simple strategy that reuses
> SRAT defined proximity domains from part of a window to fill out
> the entire window, based on the knowledge* that all of a CFMWS
> window is of a similar performance class.
>
> *Note that this heuristic will evolve when CFMWS Windows present
> a wider range of characteristics. The extension of the proximity
> domain, implemented here, is likely a step in developing a more
> sophisticated performance profile in the future.
>
> There is no change in behavior when the SRAT does not describe
> the CFMWS Window at all. In that case, a new NUMA node with a
> single memblk covering the entire CFMWS Window is created.
>
> Fixes: fd49f99c1809 ("ACPI: NUMA: Add a node and memblk for each CFMWS not in SRAT")
> Reported-by: Derick Marks <[email protected]>
> Suggested-by: Dan Williams <[email protected]>
> Signed-off-by: Alison Schofield <[email protected]>
> Tested-by: Derick Marks <[email protected]>
Reviewed-by: Dan Williams <[email protected]>