As per PAPR that defines the OS to hypervisor interface on POWER,
there is no way to calculate the node_distance between 2 nodes, when
either of the nodes are offline. However scheduler needs the distance
map to be populated at boot time. On POWER, this information is
provided within the distance_ref_points_depth array, which needs to be
parsed to extract the potential node distances.
To handle this scenario, lets overload arch_populate_distance_map(),
to provide all the distances that are possible in the current
platform.
Cc: LKML <[email protected]>
Cc: [email protected]
Cc: Nathan Lynch <[email protected]>
Cc: Michael Ellerman <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: Scott Cheloha <[email protected]>
Cc: Gautham R Shenoy <[email protected]>
Cc: Dietmar Eggemann <[email protected]>
Cc: Mel Gorman <[email protected]>
Cc: Vincent Guittot <[email protected]>
Cc: Rik van Riel <[email protected]>
Cc: Geetika Moolchandani <[email protected]>
Reported-by: Geetika Moolchandani <[email protected]>
Signed-off-by: Srikar Dronamraju <[email protected]>
---
arch/powerpc/include/asm/topology.h | 3 +++
arch/powerpc/mm/numa.c | 19 +++++++++++++++++++
2 files changed, 22 insertions(+)
diff --git a/arch/powerpc/include/asm/topology.h b/arch/powerpc/include/asm/topology.h
index e4db64c0e184..d7605d833b8d 100644
--- a/arch/powerpc/include/asm/topology.h
+++ b/arch/powerpc/include/asm/topology.h
@@ -22,6 +22,9 @@ struct drmem_lmb;
cpu_all_mask : \
node_to_cpumask_map[node])
+#define arch_populate_distance_map arch_populate_distance_map
+extern int arch_populate_distance_map(unsigned long *distance_map);
+
struct pci_bus;
#ifdef CONFIG_PCI
extern int pcibus_to_node(struct pci_bus *bus);
diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index f2bf98bdcea2..9a225b29814a 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -221,6 +221,25 @@ static void initialize_distance_lookup_table(int nid,
}
}
+int arch_populate_distance_map(unsigned long *distance_map)
+{
+ int i;
+ int distance = LOCAL_DISTANCE;
+
+ bitmap_set(distance_map, distance, 1);
+
+ if (!form1_affinity) {
+ bitmap_set(distance_map, REMOTE_DISTANCE, 1);
+ return 0;
+ }
+
+ for (i = 0; i < distance_ref_points_depth; i++) {
+ distance *= 2;
+ bitmap_set(distance_map, distance, 1);
+ }
+ return 0;
+}
+
/*
* Returns nid in the range [0..nr_node_ids], or -1 if no useful NUMA
* info is found.
--
2.27.0
On 20/05/21 21:14, Srikar Dronamraju wrote:
> +int arch_populate_distance_map(unsigned long *distance_map)
> +{
> + int i;
> + int distance = LOCAL_DISTANCE;
> +
> + bitmap_set(distance_map, distance, 1);
> +
> + if (!form1_affinity) {
> + bitmap_set(distance_map, REMOTE_DISTANCE, 1);
> + return 0;
> + }
> +
> + for (i = 0; i < distance_ref_points_depth; i++) {
> + distance *= 2;
> + bitmap_set(distance_map, distance, 1);
Do you have guarantees your distance values will always be in the form of
LOCAL_DISTANCE * 2^i
because that certainly isn't true for x86/arm64.
> + }
> + return 0;
> +}
> +
> /*
> * Returns nid in the range [0..nr_node_ids], or -1 if no useful NUMA
> * info is found.
> --
> 2.27.0
* Valentin Schneider <[email protected]> [2021-05-24 15:16:22]:
> On 20/05/21 21:14, Srikar Dronamraju wrote:
> > +int arch_populate_distance_map(unsigned long *distance_map)
> > +{
> > + int i;
> > + int distance = LOCAL_DISTANCE;
> > +
> > + bitmap_set(distance_map, distance, 1);
> > +
> > + if (!form1_affinity) {
> > + bitmap_set(distance_map, REMOTE_DISTANCE, 1);
> > + return 0;
> > + }
> > +
> > + for (i = 0; i < distance_ref_points_depth; i++) {
> > + distance *= 2;
> > + bitmap_set(distance_map, distance, 1);
>
> Do you have guarantees your distance values will always be in the form of
>
> LOCAL_DISTANCE * 2^i
>
> because that certainly isn't true for x86/arm64.
>
This is true till now. It don't think that's going to change anytime soon, but
we never know what lies ahead.
For all practical purposes, (unless a newer, shinier property is proposed,)
distance_ref_points_depth is going to give us the unique distances.
> > + }
> > + return 0;
> > +}
> > +
> > /*
> > * Returns nid in the range [0..nr_node_ids], or -1 if no useful NUMA
> > * info is found.
> > --
> > 2.27.0
--
Thanks and Regards
Srikar Dronamraju