2023-05-19 19:56:50

by Steve Wahl

[permalink] [raw]
Subject: [PATCH RESEND v5 0/8] Support sub-NUMA clustering on UV

Sub-NUMA clustering (SNC) invalidates previous assumptions of a 1:1
relationship between blades, sockets, and nodes. Make changes
necessary to support this, including some general improvements along the
way.

Steve Wahl (8):
x86/platform/uv: Add platform resolving #defines for misc
GAM_MMIOH_REDIRECT*
x86/platform/uv: Introduce helper function uv_pnode_to_socket.
x86/platform/uv: Fix incorrect mask define use in calc_mmioh_map
x86/platform/uv: When searching for minimums, start at INT_MAX not
99999
x86/platform/uv: helper functions for allocating and freeing
conversion tables
x86/platform/uv: UV support for sub-NUMA clustering
x86/platform/uv: Remove remaining BUG_ON() and BUG() calls
x86/platform/uv: Update UV[23] platform code for SNC

arch/x86/include/asm/uv/uv_hub.h | 32 ++-
arch/x86/include/asm/uv/uv_mmrs.h | 18 +-
arch/x86/kernel/apic/x2apic_uv_x.c | 318 ++++++++++++++++++-----------
3 files changed, 232 insertions(+), 136 deletions(-)

--
v2: Include the subsystem name (x86/platform/uv:) on the first line of
the commit (subject line).

v3: Use return from WARN_ON_ONCE() to simplify the code, remove memory
leaks from memory allocation failure conditions, and remove an
unnecessary set of curly braces, per comments from Ingo Molnar.

v4: Testing found that some configurations required adding a check for
"uv_blade_to_node(bid) == SOCK_EMPTY" in uv_system_init_hub().
Testing also found that some uv3 configurations didn't work correctly
with the new blade number handling. The range of blade numbers is now
needed, not just the count, so calc_mmioh_map() and
boot_init_possible_blades() needed some adjustment. And a reviewer
suggested using INT_MAX instead of 999999.

v5: Break up a single patch into smaller patches and re-factor a bit to
make the changes more understandable. Unfortunately, the fundamental
change of the relationship between nodes, pnodes, and hubs touches a
lot of places that need to be changed at the same time, so patch 6 is
still rather large.

2.26.2



2023-05-31 16:37:17

by Steve Wahl

[permalink] [raw]
Subject: Re: [PATCH RESEND v5 0/8] Support sub-NUMA clustering on UV

On Fri, May 19, 2023 at 02:07:44PM -0500, Steve Wahl wrote:
> Sub-NUMA clustering (SNC) invalidates previous assumptions of a 1:1
> relationship between blades, sockets, and nodes. Make changes
> necessary to support this, including some general improvements along the
> way.
>
> Steve Wahl (8):
> x86/platform/uv: Add platform resolving #defines for misc
> GAM_MMIOH_REDIRECT*
> x86/platform/uv: Introduce helper function uv_pnode_to_socket.
> x86/platform/uv: Fix incorrect mask define use in calc_mmioh_map
> x86/platform/uv: When searching for minimums, start at INT_MAX not
> 99999
> x86/platform/uv: helper functions for allocating and freeing
> conversion tables
> x86/platform/uv: UV support for sub-NUMA clustering
> x86/platform/uv: Remove remaining BUG_ON() and BUG() calls
> x86/platform/uv: Update UV[23] platform code for SNC
>
> arch/x86/include/asm/uv/uv_hub.h | 32 ++-
> arch/x86/include/asm/uv/uv_mmrs.h | 18 +-
> arch/x86/kernel/apic/x2apic_uv_x.c | 318 ++++++++++++++++++-----------
> 3 files changed, 232 insertions(+), 136 deletions(-)
>
> --
> v2: Include the subsystem name (x86/platform/uv:) on the first line of
> the commit (subject line).
>
> v3: Use return from WARN_ON_ONCE() to simplify the code, remove memory
> leaks from memory allocation failure conditions, and remove an
> unnecessary set of curly braces, per comments from Ingo Molnar.
>
> v4: Testing found that some configurations required adding a check for
> "uv_blade_to_node(bid) == SOCK_EMPTY" in uv_system_init_hub().
> Testing also found that some uv3 configurations didn't work correctly
> with the new blade number handling. The range of blade numbers is now
> needed, not just the count, so calc_mmioh_map() and
> boot_init_possible_blades() needed some adjustment. And a reviewer
> suggested using INT_MAX instead of 999999.
>
> v5: Break up a single patch into smaller patches and re-factor a bit to
> make the changes more understandable. Unfortunately, the fundamental
> change of the relationship between nodes, pnodes, and hubs touches a
> lot of places that need to be changed at the same time, so patch 6 is
> still rather large.

Gentle ping. We would like to get this in before the hardware it
supports (Sapphire Rapids based) ships. If there's some other avenue
I should be taking, please let me know!

Thanks,

--> Steve Wahl, [email protected]