Date: Mon, 8 Dec 2014 11:08:04 -0800
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: David Hildenbrand <dahi@linux.vnet.ibm.com>
Cc: linux-kernel@vger.kernel.org, heiko.carstens@de.ibm.com,
        borntraeger@de.ibm.com, rafael.j.wysocki@intel.com,
        peterz@infradead.org, srivatsa.bhat@linux.vnet.ibm.com,
        oleg@redhat.com, bp@suse.de, jkosina@suse.cz
Subject: Re: [PATCH v1] CPU hotplug: active_reader not woken up in some cases
 - deadlock
Message-ID: <20141208190803.GR25340@linux.vnet.ibm.com>
Reply-To: paulmck@linux.vnet.ibm.com
References: <1418062383-39657-1-git-send-email-dahi@linux.vnet.ibm.com>
 <20141208183134.GQ25340@linux.vnet.ibm.com>
 <20141208195814.663007dd@thinkpad-w530>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20141208195814.663007dd@thinkpad-w530>
User-Agent: Mutt/1.5.21 (2010-09-15)
Sender: linux-kernel-owner@vger.kernel.org

On Mon, Dec 08, 2014 at 07:58:14PM +0100, David Hildenbrand wrote:
> > On Mon, Dec 08, 2014 at 07:13:03PM +0100, David Hildenbrand wrote:
> > > Commit b2c4623dcd07 ("rcu: More on deadlock between CPU hotplug and expedited
> > > grace periods") introduced another problem that can easily be reproduced by
> > > starting/stopping cpus in a loop.
> > > 
> > > E.g.:
> > >   for i in `seq 5000`; do
> > >       echo 1 > /sys/devices/system/cpu/cpu1/online
> > >       echo 0 > /sys/devices/system/cpu/cpu1/online
> > >   done
> > > 
> > > Will result in:
> > >   INFO: task /cpu_start_stop:1 blocked for more than 120 seconds.
> > >   Call Trace:
> > >   ([<00000000006a028e>] __schedule+0x406/0x91c)
> > >    [<0000000000130f60>] cpu_hotplug_begin+0xd0/0xd4
> > >    [<0000000000130ff6>] _cpu_up+0x3e/0x1c4
> > >    [<0000000000131232>] cpu_up+0xb6/0xd4
> > >    [<00000000004a5720>] device_online+0x80/0xc0
> > >    [<00000000004a57f0>] online_store+0x90/0xb0
> > >   ...
> > > 
> > > And a deadlock.
> > > 
> > > Problem is that if the last ref in put_online_cpus() can't get the
> > > cpu_hotplug.lock the puts_pending count is incremented, but a sleeping active_writer
> > > might never be woken up, therefore never exiting the loop in cpu_hotplug_begin().
> > > 
> > > This quick fix wakes up the active_writer proactively. The writer already
> > > goes back to sleep if the ref count isn't already down to 0, so this should be
> > > fine.
> > > 
> > > Can't reproduce the error with this fix.
> > 
> > Good catch!
> > 
> > But don't we need to use exactly the same value for the NULL check
> > and for the wakeup?  Otherwise, wouldn't it be possible for
> > cpu_hotplug.active_writer to be non-NULL for the check but NULL
> > for the wake_up_process()?
> > 
> > 							Thanx, Paul
> 
> active_writer is cleared while holding cpuhp_lock, so this should be safe,
> right?

You lost me on that one.  Don't we get to that piece of code precisely
because we don't hold any of the CPU-hotplug locks?  If so, the
writer might well hold all the locks it needs, and might well change
cpu_hotplug.active_writer out from under us.

What am I missing here?

							Thanx, Paul

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/