The call to rcu_cpu_starting() in start_secondary() is not early enough
in the CPU-hotplug onlining process, which results in lockdep splats as
follows:
WARNING: suspicious RCU usage
-----------------------------
kernel/locking/lockdep.c:3497 RCU-list traversed in non-reader section!!
other info that might help us debug this:
RCU used illegally from offline CPU!
rcu_scheduler_active = 1, debug_locks = 1
no locks held by swapper/1/0.
Call Trace:
dump_stack+0xec/0x144 (unreliable)
lockdep_rcu_suspicious+0x128/0x14c
__lock_acquire+0x1060/0x1c60
lock_acquire+0x140/0x5f0
_raw_spin_lock_irqsave+0x64/0xb0
clockevents_register_device+0x74/0x270
register_decrementer_clockevent+0x94/0x110
start_secondary+0x134/0x800
start_secondary_prolog+0x10/0x14
This is avoided by moving the call to rcu_cpu_starting up near the
beginning of the start_secondary() function. Note that the
raw_smp_processor_id() is required in order to avoid calling into
lockdep before RCU has declared the CPU to be watched for readers.
Link: https://lore.kernel.org/lkml/160223032121.7002.1269740091547117869.tip-bot2@tip-bot2/
Signed-off-by: Qian Cai <[email protected]>
---
arch/powerpc/kernel/smp.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index 3c6b9822f978..8c2857cbd960 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -1393,13 +1393,14 @@ static void add_cpu_to_masks(int cpu)
/* Activate a secondary processor. */
void start_secondary(void *unused)
{
- unsigned int cpu = smp_processor_id();
+ unsigned int cpu = raw_smp_processor_id();
mmgrab(&init_mm);
current->active_mm = &init_mm;
smp_store_cpu_info(cpu);
set_dec(tb_ticks_per_jiffy);
+ rcu_cpu_starting(cpu);
preempt_disable();
cpu_callin_map[cpu] = 1;
--
2.28.0
Qian Cai <[email protected]> writes:
> The call to rcu_cpu_starting() in start_secondary() is not early enough
> in the CPU-hotplug onlining process, which results in lockdep splats as
> follows:
Since when?
What kernel version?
I haven't seen this running CPU hotplug tests with PROVE_LOCKING=y on
v5.10-rc1. Am I missing a CONFIG?
cheers
> WARNING: suspicious RCU usage
> -----------------------------
> kernel/locking/lockdep.c:3497 RCU-list traversed in non-reader section!!
>
> other info that might help us debug this:
>
> RCU used illegally from offline CPU!
> rcu_scheduler_active = 1, debug_locks = 1
> no locks held by swapper/1/0.
>
> Call Trace:
> dump_stack+0xec/0x144 (unreliable)
> lockdep_rcu_suspicious+0x128/0x14c
> __lock_acquire+0x1060/0x1c60
> lock_acquire+0x140/0x5f0
> _raw_spin_lock_irqsave+0x64/0xb0
> clockevents_register_device+0x74/0x270
> register_decrementer_clockevent+0x94/0x110
> start_secondary+0x134/0x800
> start_secondary_prolog+0x10/0x14
>
> This is avoided by moving the call to rcu_cpu_starting up near the
> beginning of the start_secondary() function. Note that the
> raw_smp_processor_id() is required in order to avoid calling into
> lockdep before RCU has declared the CPU to be watched for readers.
>
> Link: https://lore.kernel.org/lkml/160223032121.7002.1269740091547117869.tip-bot2@tip-bot2/
> Signed-off-by: Qian Cai <[email protected]>
> ---
> arch/powerpc/kernel/smp.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
> index 3c6b9822f978..8c2857cbd960 100644
> --- a/arch/powerpc/kernel/smp.c
> +++ b/arch/powerpc/kernel/smp.c
> @@ -1393,13 +1393,14 @@ static void add_cpu_to_masks(int cpu)
> /* Activate a secondary processor. */
> void start_secondary(void *unused)
> {
> - unsigned int cpu = smp_processor_id();
> + unsigned int cpu = raw_smp_processor_id();
>
> mmgrab(&init_mm);
> current->active_mm = &init_mm;
>
> smp_store_cpu_info(cpu);
> set_dec(tb_ticks_per_jiffy);
> + rcu_cpu_starting(cpu);
> preempt_disable();
> cpu_callin_map[cpu] = 1;
>
> --
> 2.28.0
On Thu, Oct 29, 2020 at 11:09:07AM +1100, Michael Ellerman wrote:
> Qian Cai <[email protected]> writes:
> > The call to rcu_cpu_starting() in start_secondary() is not early enough
> > in the CPU-hotplug onlining process, which results in lockdep splats as
> > follows:
>
> Since when?
> What kernel version?
>
> I haven't seen this running CPU hotplug tests with PROVE_LOCKING=y on
> v5.10-rc1. Am I missing a CONFIG?
My guess would be that adding CONFIG_PROVE_RAW_LOCK_NESTING=y will
get you some splats.
Thanx, Paul
> cheers
>
>
> > WARNING: suspicious RCU usage
> > -----------------------------
> > kernel/locking/lockdep.c:3497 RCU-list traversed in non-reader section!!
> >
> > other info that might help us debug this:
> >
> > RCU used illegally from offline CPU!
> > rcu_scheduler_active = 1, debug_locks = 1
> > no locks held by swapper/1/0.
> >
> > Call Trace:
> > dump_stack+0xec/0x144 (unreliable)
> > lockdep_rcu_suspicious+0x128/0x14c
> > __lock_acquire+0x1060/0x1c60
> > lock_acquire+0x140/0x5f0
> > _raw_spin_lock_irqsave+0x64/0xb0
> > clockevents_register_device+0x74/0x270
> > register_decrementer_clockevent+0x94/0x110
> > start_secondary+0x134/0x800
> > start_secondary_prolog+0x10/0x14
> >
> > This is avoided by moving the call to rcu_cpu_starting up near the
> > beginning of the start_secondary() function. Note that the
> > raw_smp_processor_id() is required in order to avoid calling into
> > lockdep before RCU has declared the CPU to be watched for readers.
> >
> > Link: https://lore.kernel.org/lkml/160223032121.7002.1269740091547117869.tip-bot2@tip-bot2/
> > Signed-off-by: Qian Cai <[email protected]>
> > ---
> > arch/powerpc/kernel/smp.c | 3 ++-
> > 1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
> > index 3c6b9822f978..8c2857cbd960 100644
> > --- a/arch/powerpc/kernel/smp.c
> > +++ b/arch/powerpc/kernel/smp.c
> > @@ -1393,13 +1393,14 @@ static void add_cpu_to_masks(int cpu)
> > /* Activate a secondary processor. */
> > void start_secondary(void *unused)
> > {
> > - unsigned int cpu = smp_processor_id();
> > + unsigned int cpu = raw_smp_processor_id();
> >
> > mmgrab(&init_mm);
> > current->active_mm = &init_mm;
> >
> > smp_store_cpu_info(cpu);
> > set_dec(tb_ticks_per_jiffy);
> > + rcu_cpu_starting(cpu);
> > preempt_disable();
> > cpu_callin_map[cpu] = 1;
> >
> > --
> > 2.28.0
On Wed, Oct 28, 2020 at 02:23:34PM -0400, Qian Cai wrote:
> The call to rcu_cpu_starting() in start_secondary() is not early enough
> in the CPU-hotplug onlining process, which results in lockdep splats as
> follows:
>
> WARNING: suspicious RCU usage
> -----------------------------
> kernel/locking/lockdep.c:3497 RCU-list traversed in non-reader section!!
>
> other info that might help us debug this:
>
> RCU used illegally from offline CPU!
> rcu_scheduler_active = 1, debug_locks = 1
> no locks held by swapper/1/0.
>
> Call Trace:
> dump_stack+0xec/0x144 (unreliable)
> lockdep_rcu_suspicious+0x128/0x14c
> __lock_acquire+0x1060/0x1c60
> lock_acquire+0x140/0x5f0
> _raw_spin_lock_irqsave+0x64/0xb0
> clockevents_register_device+0x74/0x270
> register_decrementer_clockevent+0x94/0x110
> start_secondary+0x134/0x800
> start_secondary_prolog+0x10/0x14
>
> This is avoided by moving the call to rcu_cpu_starting up near the
> beginning of the start_secondary() function. Note that the
> raw_smp_processor_id() is required in order to avoid calling into
> lockdep before RCU has declared the CPU to be watched for readers.
>
> Link: https://lore.kernel.org/lkml/160223032121.7002.1269740091547117869.tip-bot2@tip-bot2/
> Signed-off-by: Qian Cai <[email protected]>
Acked-by: Paul E. McKenney <[email protected]>
> ---
> arch/powerpc/kernel/smp.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
> index 3c6b9822f978..8c2857cbd960 100644
> --- a/arch/powerpc/kernel/smp.c
> +++ b/arch/powerpc/kernel/smp.c
> @@ -1393,13 +1393,14 @@ static void add_cpu_to_masks(int cpu)
> /* Activate a secondary processor. */
> void start_secondary(void *unused)
> {
> - unsigned int cpu = smp_processor_id();
> + unsigned int cpu = raw_smp_processor_id();
>
> mmgrab(&init_mm);
> current->active_mm = &init_mm;
>
> smp_store_cpu_info(cpu);
> set_dec(tb_ticks_per_jiffy);
> + rcu_cpu_starting(cpu);
> preempt_disable();
> cpu_callin_map[cpu] = 1;
>
> --
> 2.28.0
>
On Thu, 2020-10-29 at 11:09 +1100, Michael Ellerman wrote:
> Qian Cai <[email protected]> writes:
> > The call to rcu_cpu_starting() in start_secondary() is not early enough
> > in the CPU-hotplug onlining process, which results in lockdep splats as
> > follows:
>
> Since when?
For me, it is since the commit in the link which looks now merged into
v5.10-rc1. Then, it needs CONFIG_PROVE_RCU_LIST=y.
> What kernel version?
>
> I haven't seen this running CPU hotplug tests with PROVE_LOCKING=y on
> v5.10-rc1. Am I missing a CONFIG?
>
> cheers
>
>
> > WARNING: suspicious RCU usage
> > -----------------------------
> > kernel/locking/lockdep.c:3497 RCU-list traversed in non-reader section!!
> >
> > other info that might help us debug this:
> >
> > RCU used illegally from offline CPU!
> > rcu_scheduler_active = 1, debug_locks = 1
> > no locks held by swapper/1/0.
> >
> > Call Trace:
> > dump_stack+0xec/0x144 (unreliable)
> > lockdep_rcu_suspicious+0x128/0x14c
> > __lock_acquire+0x1060/0x1c60
> > lock_acquire+0x140/0x5f0
> > _raw_spin_lock_irqsave+0x64/0xb0
> > clockevents_register_device+0x74/0x270
> > register_decrementer_clockevent+0x94/0x110
> > start_secondary+0x134/0x800
> > start_secondary_prolog+0x10/0x14
> >
> > This is avoided by moving the call to rcu_cpu_starting up near the
> > beginning of the start_secondary() function. Note that the
> > raw_smp_processor_id() is required in order to avoid calling into
> > lockdep before RCU has declared the CPU to be watched for readers.
> >
> > Link:
> > https://lore.kernel.org/lkml/160223032121.7002.1269740091547117869.tip-bot2@tip-bot2/
> > Signed-off-by: Qian Cai <[email protected]>
> > ---
> > arch/powerpc/kernel/smp.c | 3 ++-
> > 1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
> > index 3c6b9822f978..8c2857cbd960 100644
> > --- a/arch/powerpc/kernel/smp.c
> > +++ b/arch/powerpc/kernel/smp.c
> > @@ -1393,13 +1393,14 @@ static void add_cpu_to_masks(int cpu)
> > /* Activate a secondary processor. */
> > void start_secondary(void *unused)
> > {
> > - unsigned int cpu = smp_processor_id();
> > + unsigned int cpu = raw_smp_processor_id();
> >
> > mmgrab(&init_mm);
> > current->active_mm = &init_mm;
> >
> > smp_store_cpu_info(cpu);
> > set_dec(tb_ticks_per_jiffy);
> > + rcu_cpu_starting(cpu);
> > preempt_disable();
> > cpu_callin_map[cpu] = 1;
> >
> > --
> > 2.28.0
On Wed, 2020-10-28 at 17:31 -0700, Paul E. McKenney wrote:
> On Thu, Oct 29, 2020 at 11:09:07AM +1100, Michael Ellerman wrote:
> > Qian Cai <[email protected]> writes:
> > > The call to rcu_cpu_starting() in start_secondary() is not early enough
> > > in the CPU-hotplug onlining process, which results in lockdep splats as
> > > follows:
> >
> > Since when?
> > What kernel version?
> >
> > I haven't seen this running CPU hotplug tests with PROVE_LOCKING=y on
> > v5.10-rc1. Am I missing a CONFIG?
>
> My guess would be that adding CONFIG_PROVE_RAW_LOCK_NESTING=y will
> get you some splats.
Well, I don't have that set, so it should be CONFIG_PROVE_RCU_LIST=y. Anyway,
this is .config to reproduce on Power9 NV:
https://cailca.coding.net/public/linux/mm/git/files/master/powerpc.config
On Wed, 28 Oct 2020 14:23:34 -0400, Qian Cai wrote:
> The call to rcu_cpu_starting() in start_secondary() is not early enough
> in the CPU-hotplug onlining process, which results in lockdep splats as
> follows:
>
> WARNING: suspicious RCU usage
> -----------------------------
> kernel/locking/lockdep.c:3497 RCU-list traversed in non-reader section!!
>
> [...]
Applied to powerpc/fixes.
[1/1] powerpc/smp: Call rcu_cpu_starting() earlier
https://git.kernel.org/powerpc/c/99f070b62322a4b8c1252952735806d09eb44b68
cheers