Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <20190626135447.y24mvfuid5fifwjc@linutronix.de>
 <20190626162558.GY26519@linux.ibm.com> <20190627142436.GD215968@google.com>
 <20190627103455.01014276@gandalf.local.home> <20190627153031.GA249127@google.com>
 <CAEXW_YT5LgdP_9SrachU4ZrhV9a7o_DM8eBfgxj=n7yRRyS-TQ@mail.gmail.com> <20190627154011.vbje64x6auaknhx4@linutronix.de>
In-Reply-To: <20190627154011.vbje64x6auaknhx4@linutronix.de>
From:   Joel Fernandes <joel@joelfernandes.org>
Date:   Thu, 27 Jun 2019 13:43:54 -0400
Message-ID: <CAEXW_YTvkSTqwi_jOE2Pr+uD-GC4Xv0CtBEL9YO7=LvJcM3FBQ@mail.gmail.com>
Subject: Re: [RFC] Deadlock via recursive wakeup via RCU with threadirqs
To:     Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc:     Steven Rostedt <rostedt@goodmis.org>,
        "Paul E. McKenney" <paulmck@linux.ibm.com>,
        rcu <rcu@vger.kernel.org>, LKML <linux-kernel@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Josh Triplett <josh@joshtriplett.org>,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        Lai Jiangshan <jiangshanlai@gmail.com>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Thu, Jun 27, 2019 at 11:40 AM Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
>
> On 2019-06-27 11:37:10 [-0400], Joel Fernandes wrote:
> > Sebastian it would be nice if possible to trace where the
> > t->rcu_read_unlock_special is set for this scenario of calling
> > rcu_read_unlock_special, to give a clear idea about whether it was
> > really because of an IPI. I guess we could also add additional RCU
> > debug fields to task_struct (just for debugging) to see where there
> > unlock_special is set.
> >
> > Is there a test to reproduce this, or do I just boot an intel x86_64
> > machine with "threadirqs" and run into it?
>
> Do you want to send me a patch or should I send you my kvm image which
> triggers the bug on boot?

I could reproduce this as well just booting Linus tree with threadirqs
command line and running rcutorture. In 15 seconds or so it locks
up... gdb backtrace shows the recursive lock:

(gdb) bt
#0  queued_spin_lock_slowpath (lock=0xffff88813af16b44, val=0) at
kernel/locking/qspinlock.c:381
#1  0xffffffff81a75a0f in queued_spin_lock (lock=<optimized out>) at
./include/asm-generic/qspinlock.h:81
#2  do_raw_spin_lock_flags (flags=<optimized out>, lock=<optimized
out>) at ./include/linux/spinlock.h:193
#3  __raw_spin_lock_irqsave (lock=<optimized out>) at
./include/linux/spinlock_api_smp.h:119
#4  _raw_spin_lock_irqsave (lock=0xffff88813af16b44) at
kernel/locking/spinlock.c:159
#5  0xffffffff81092158 in try_to_wake_up (p=0xffff88813af16400,
state=3, wake_flags=0) at kernel/sched/core.c:2000
#6  0xffffffff8109265c in wake_up_process (p=<optimized out>) at
kernel/sched/core.c:2114
#7  0xffffffff8106b019 in wakeup_softirqd () at kernel/softirq.c:76
#8  0xffffffff8106b8ef in raise_softirq_irqoff (nr=<optimized out>) at
kernel/softirq.c:437
#9  0xffffffff810d080d in rcu_read_unlock_special (t=<optimized out>)
at kernel/rcu/tree_plugin.h:632
#10 0xffffffff810d0868 in __rcu_read_unlock () at kernel/rcu/tree_plugin.h:414
#11 0xffffffff810aa58e in rcu_read_unlock () at ./include/linux/rcupdate.h:646
#12 cpuacct_charge (tsk=<optimized out>, cputime=<optimized out>) at
kernel/sched/cpuacct.c:352
#13 0xffffffff81097994 in cgroup_account_cputime
(delta_exec=<optimized out>, task=<optimized out>) at
./include/linux/cgroup.h:764
#14 update_curr (cfs_rq=<optimized out>) at kernel/sched/fair.c:843
#15 0xffffffff8109a882 in enqueue_entity (flags=<optimized out>,
se=<optimized out>, cfs_rq=<optimized out>) at
kernel/sched/fair.c:3901
#16 enqueue_task_fair (rq=<optimized out>, p=<optimized out>, flags=9)
at kernel/sched/fair.c:5194
#17 0xffffffff81091699 in enqueue_task (flags=<optimized out>,
p=<optimized out>, rq=<optimized out>) at kernel/sched/core.c:774
#18 activate_task (rq=0xffff88813af16b44, p=0x0 <fixed_percpu_data>,
flags=<optimized out>) at kernel/sched/core.c:795
#19 0xffffffff81091a1a in ttwu_do_activate (rq=0xffff88813af16b44,
p=0x0 <fixed_percpu_data>, wake_flags=0, rf=<optimized out>) at
kernel/sched/core.c:1737
#20 0xffffffff810924b3 in ttwu_queue (wake_flags=<optimized out>,
cpu=<optimized out>, p=<optimized out>) at kernel/sched/core.c:1882
#21 try_to_wake_up (p=0xffff88813af16400, state=<optimized out>,
wake_flags=0) at kernel/sched/core.c:2092
#22 0xffffffff8109265c in wake_up_process (p=<optimized out>) at
kernel/sched/core.c:2114
#23 0xffffffff8106b019 in wakeup_softirqd () at kernel/softirq.c:76
#24 0xffffffff8106b843 in invoke_softirq () at kernel/softirq.c:383
#25 irq_exit () at kernel/softirq.c:413
#26 0xffffffff81c01f8e in exiting_irq () at ./arch/x86/include/asm/apic.h:536


>
> Sebastian