2012-11-29 00:04:56

by Dave Hansen

[permalink] [raw]
Subject: 32/64-bit NUMA consolidation behavior regresion

Hi Tejun,

I was bisecting a boot problem on a 32-bit NUMA kernel and it bisected
down to commit 8db78cc4. It turns out that, with this patch,
pcpu_need_numa() changed its return value on my system from 1 to 0.
What that basically meant was that we stopped using the remapped lowmem
areas for percpu data.

My system is just qemu booted with:

-smp 8 -m 8192 -numa node,nodeid=0,cpus=0-3 -numa node,nodeid=1,cpus=4-7

Watch the "PERCPU:" line early in boot, and you can see the "Embedded"
come and go with or without your patch:

[ 0.000000] PERCPU: Embedded 11 pages/cpu @f3000000 s30592 r0 d14464
vs
[ 0.000000] PERCPU: 11 4K pages/cpu @f83fe000 s30592 r0 d14464

I believe this has to do with the hunks in your patch that do:

-#ifdef CONFIG_X86_64
init_cpu_to_node();
-#endif
...
-#ifdef CONFIG_X86_32
-DEFINE_EARLY_PER_CPU(int, x86_cpu_to_node_map, 0);
-#else
DEFINE_EARLY_PER_CPU(int, x86_cpu_to_node_map, NUMA_NO_NODE);
-#endif
EXPORT_EARLY_PER_CPU_SYMBOL(x86_cpu_to_node_map);

I don't have a fix handy because I'm working on the original problem,
but I just happened to run across this during a bisect.


2012-11-30 20:42:49

by Tejun Heo

[permalink] [raw]
Subject: Re: 32/64-bit NUMA consolidation behavior regresion

Hello, Dave.

On Wed, Nov 28, 2012 at 04:03:58PM -0800, Dave Hansen wrote:
> My system is just qemu booted with:
>
> -smp 8 -m 8192 -numa node,nodeid=0,cpus=0-3 -numa node,nodeid=1,cpus=4-7
>
> Watch the "PERCPU:" line early in boot, and you can see the "Embedded"
> come and go with or without your patch:
>
> [ 0.000000] PERCPU: Embedded 11 pages/cpu @f3000000 s30592 r0 d14464
> vs
> [ 0.000000] PERCPU: 11 4K pages/cpu @f83fe000 s30592 r0 d14464
...
> I don't have a fix handy because I'm working on the original problem,
> but I just happened to run across this during a bisect.

Just tested 3.7-rc7 w/ qemu and it works as expected here.

Can you please boot with the following debug patch and report the boot
message before and after?

Thanks.

diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c
index 5cdff03..1133dc8 100644
--- a/arch/x86/kernel/setup_percpu.c
+++ b/arch/x86/kernel/setup_percpu.c
@@ -71,9 +71,13 @@ static bool __init pcpu_need_numa(void)
for_each_possible_cpu(cpu) {
int node = early_cpu_to_node(cpu);

+ printk("XXX pcpu_need_numa: cpu%d@%d online=%d ND=%p\n",
+ cpu, node, node_online(node), NODE_DATA(node));
if (node_online(node) && NODE_DATA(node) &&
- last && last != NODE_DATA(node))
+ last && last != NODE_DATA(node)) {
+ printk("XXX need numa\n");
return true;
+ }

last = NODE_DATA(node);
}

2012-11-30 20:55:58

by Dave Hansen

[permalink] [raw]
Subject: Re: 32/64-bit NUMA consolidation behavior regresion

On 11/30/2012 12:42 PM, Tejun Heo wrote:
> On Wed, Nov 28, 2012 at 04:03:58PM -0800, Dave Hansen wrote:
>> My system is just qemu booted with:
>>
>> -smp 8 -m 8192 -numa node,nodeid=0,cpus=0-3 -numa node,nodeid=1,cpus=4-7
>>
>> Watch the "PERCPU:" line early in boot, and you can see the "Embedded"
>> come and go with or without your patch:
>>
>> [ 0.000000] PERCPU: Embedded 11 pages/cpu @f3000000 s30592 r0 d14464
>> vs
>> [ 0.000000] PERCPU: 11 4K pages/cpu @f83fe000 s30592 r0 d14464
> ...
>> I don't have a fix handy because I'm working on the original problem,
>> but I just happened to run across this during a bisect.
>
> Just tested 3.7-rc7 w/ qemu and it works as expected here.
>
> Can you please boot with the following debug patch and report the boot
> message before and after?

Hi Tejun,

I just tested with 3.7-rc7 and I'm seeing the expected behavior now.
Looks like it got fixed along the way somewhere. I was bisecting way
back in the 2.6.3x's. Sorry of the noise.