Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1751947AbdIEOvD (ORCPT <rfc822;w@1wt.eu>);
        Tue, 5 Sep 2017 10:51:03 -0400
Received: from mx1.redhat.com ([209.132.183.28]:45552 "EHLO mx1.redhat.com"
        rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
        id S1751732AbdIEOvC (ORCPT <rfc822;linux-kernel@vger.kernel.org>);
        Tue, 5 Sep 2017 10:51:02 -0400
DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 150B0C014199
Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com
Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dzickus@redhat.com
Date: Tue, 5 Sep 2017 10:51:00 -0400
From: Don Zickus <dzickus@redhat.com>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: LKML <linux-kernel@vger.kernel.org>,
        Peter Zijlstra <peterz@infradead.org>, Ingo Molnar <mingo@kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Borislav Petkov <bp@alien8.de>,
        Sebastian Siewior <bigeasy@linutronix.de>,
        Nicholas Piggin <npiggin@gmail.com>,
        Chris Metcalf <cmetcalf@mellanox.com>,
        Ulrich Obergfell <uobergfe@redhat.com>
Subject: Re: [patch 10/29] lockup_detector/perf: Prevent cpu hotplug deadlock
Message-ID: <20170905145100.fudvvm3udlcwqioh@redhat.com>
References: <20170831071558.995235362@linutronix.de>
 <20170831073053.770526691@linutronix.de>
 <20170901190208.pn4vq25udylxehph@redhat.com>
 <alpine.DEB.2.20.1709012127430.2372@nanos>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <alpine.DEB.2.20.1709012127430.2372@nanos>
User-Agent: NeoMutt/20170428-dirty (1.8.2)
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Tue, 05 Sep 2017 14:51:02 +0000 (UTC)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 1772
Lines: 44

On Fri, Sep 01, 2017 at 09:29:07PM +0200, Thomas Gleixner wrote:
> On Fri, 1 Sep 2017, Don Zickus wrote:
> > On Thu, Aug 31, 2017 at 09:16:08AM +0200, Thomas Gleixner wrote:
> > > The following deadlock is possible in the watchdog hotplug code:
> > > 
> > >   cpus_write_lock()
> > >     ...
> > >       takedown_cpu()
> > >         smpboot_park_threads()
> > >           smpboot_park_thread()
> > >             kthread_park()
> > >               ->park() := watchdog_disable()
> > >                 watchdog_nmi_disable()
> > >                   perf_event_release_kernel();
> > >                     put_event()
> > >                       _free_event()
> > >                         ->destroy() := hw_perf_event_destroy()
> > >                           x86_release_hardware()
> > >                             release_ds_buffers()
> > >                               get_online_cpus()
> > > 
> > > when a per cpu watchdog perf event is destroyed which drops the last
> > > reference to the PMU hardware. The cleanup code there invokes
> > > get_online_cpus() which instantly deadlocks because the hotplug percpu
> > > rwsem is write locked.
> > 
> > The main reason perf_event_release_kernel is in this path is because the
> > oprofile folks complained they couldn't use the perf counters when the
> > nmi_watchdog was disabled on the command line.
> 
> If the nmi watchdog is disabled on the command line then there are no
> counters claimed at all.

Ah, I see it now.  When you park all the threads, you clear the cpumask
which then calls the lockup detector_cleanup to release all perf_counters
that are no longer used by the cpumask.

Further reading this code, I see how the code covers various race conditions
we tried solving in the past.

Thanks!

Cheers,
Don