2013-03-07 02:49:15

by Ming Lei

Subject: [RFC PATCH] ARM: keep __my_cpu_offset consistent with generic one

Commit 14318efb(ARM: 7587/1: implement optimized percpu variable access)
introduces arm's __my_cpu_offset to optimize percpu vaiable access,
which really works well on hackbench test, but may cause __my_cpu_offset
to return garbage value before it is initialized in cpu_init() called
by setup_arch, so accessing a percpu variable before setup_arch may cause
kernel hang. But the generic__my_cpu_offset always returns zero before
percpu area is brought up.

So the patch tries to clear __my_cpu_offset on boot CPU early
to avoid boot hang.

At least now percpu variable is accessed by lockdep before
setup_arch(), and enabling CONFIG_LOCK_STAT or CONFIG_DEBUG_LOCKDEP
can trigger kernel hang.

Cc: Tejun Heo <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Rob Herring <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Nicolas Pitre <[email protected]>
Cc: Russell King <[email protected]>
Signed-off-by: Ming Lei <[email protected]>
---
arch/arm/kernel/setup.c | 7 +++++++
1 file changed, 7 insertions(+)

diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c
index 3f6cbb2..bc7ee4b 100644
--- a/arch/arm/kernel/setup.c
+++ b/arch/arm/kernel/setup.c
@@ -439,6 +439,13 @@ void __init smp_setup_processor_id(void)
for (i = 1; i < nr_cpu_ids; ++i)
cpu_logical_map(i) = i == cpu ? 0 : i;

+ /*
+ * clear __my_cpu_offset on boot CPU early to avoid hang
+ * caused by accessing percpu variable before percpu area is
+ * brought up
+ */
+ set_my_cpu_offset(0);
+
printk(KERN_INFO "Booting Linux on physical CPU 0x%x\n", mpidr);
}

--
1.7.9.5

2013-03-07 03:45:41

by Will Deacon

Subject: Re: [RFC PATCH] ARM: keep __my_cpu_offset consistent with generic one

Hello,

On Thu, Mar 07, 2013 at 02:47:48AM +0000, Ming Lei wrote:
> Commit 14318efb(ARM: 7587/1: implement optimized percpu variable access)
> introduces arm's __my_cpu_offset to optimize percpu vaiable access,
> which really works well on hackbench test, but may cause __my_cpu_offset
> to return garbage value before it is initialized in cpu_init() called
> by setup_arch, so accessing a percpu variable before setup_arch may cause
> kernel hang. But the generic__my_cpu_offset always returns zero before
> percpu area is brought up.
>
> So the patch tries to clear __my_cpu_offset on boot CPU early
> to avoid boot hang.
>
> At least now percpu variable is accessed by lockdep before
> setup_arch(), and enabling CONFIG_LOCK_STAT or CONFIG_DEBUG_LOCKDEP
> can trigger kernel hang.

For the record, can you include a backtrace or something please? The
description makes it sounds like a caller bug, so it would be good to
document a valid user of per-cpu before cpu_init().

Will

2013-03-07 04:00:17

by Ming Lei

Subject: Re: [RFC PATCH] ARM: keep __my_cpu_offset consistent with generic one

Hi,

On Thu, Mar 7, 2013 at 11:44 AM, Will Deacon <[email protected]> wrote:
>
> For the record, can you include a backtrace or something please? The

lock_release_holdtime() will access percpu variable, so every lock
release may trigger the hang if CONFIG_LOCK_STAT is enabled,
and there are many lock acquire/release before setup_arch.

No detailed backtrace since only early_printk can work at that time.

> description makes it sounds like a caller bug, so it would be good to
> document a valid user of per-cpu before cpu_init().

OK, I will add the document: lockdep will use percpu variable
before cpu_init() called inside setup_arch().

Thanks,
--
Ming Lei