Date: Wed, 3 Dec 2014 14:59:58 -0500
From: Chris Mason <clm@fb.com>
Subject: Re: frequent lockups in 3.18rc4
To: Dave Jones <davej@redhat.com>
CC: Linus Torvalds <torvalds@linux-foundation.org>,
        Mike Galbraith <umgwanakikbuti@gmail.com>,
        Ingo Molnar <mingo@kernel.org>, Peter Zijlstra <peterz@infradead.org>,
        =?iso-8859-1?q?D=E2niel?= Fraga <fragabr@gmail.com>,
        Sasha Levin <sasha.levin@oracle.com>,
        "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Message-ID: <1417636798.11261.1@mail.thefacebook.com>
In-Reply-To: <20141203190045.GB32005@redhat.com>
References: <547bbe36.48548c0a.105c.779c@mx.google.com>
	<20141201191431.GA17385@linux.vnet.ibm.com>
	<547ccf74.a5198c0a.25de.26d9@mx.google.com>
	<CA+55aFx0K=T8yvfRCYeaJG_5YO+Qm8EEQfA9Xp59xedbzj=f2A@mail.gmail.com>
	<20141201230339.GA20487@ret.masoncoding.com>
	<1417529606.3924.26.camel@maggy.simpson.net>
	<CA+55aFw8smHBw9HiCiYL_ohkULLeunWo6qfayM19zhF1hKAxXg@mail.gmail.com>
	<1417540493.21136.3@mail.thefacebook.com>
	<20141203184111.GA32005@redhat.com>
	<CA+55aFzLprvtdLGDXgRr=k3QqO824uQSzbxT-b4vu_4pryMtSA@mail.gmail.com>
	<20141203190045.GB32005@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"; format=flowed
Sender: linux-kernel-owner@vger.kernel.org


On Wed, Dec 3, 2014 at 2:00 PM, Dave Jones <davej@redhat.com> wrote:
> On Wed, Dec 03, 2014 at 10:45:57AM -0800, Linus Torvalds wrote:
>  > On Wed, Dec 3, 2014 at 10:41 AM, Dave Jones <davej@redhat.com> 
> wrote:
>  > >
>  > > I've been stuck on this kernel for a few days now trying to 
> prove it
>  > > good/bad one way or the other, and I'm leaning towards good, 
> given
>  > > that it recovers, even though the traces look similar.
>  >
>  > Ugh. But this does *not* happen with 3.16, right? Even the 
> non-fatal case?
> 
> correct. at least not in any of the runs that I did to date.
> 
>  > If so, I'd be inclined to call it "bad". But there might well be 
> two
>  > bugs: one that makes that NMI watchdog trigger, and another one 
> that
>  > then makes it be a hard lockup. I'd think it would be good to 
> figure
>  > out the "NMI watchdog starts triggering" one first, though.
> 
> I think you're right.
> 
> So right after sending my last mail, I rebooted, and restarted the run
> on the same kernel again.
> 
> As I was writing this mail, this happened.
> 
> [  524.420897] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! 
> [trinity-c178:20182]
> 
> and that's all that made it over the console. I couldn't log in via 
> ssh,
> and thought "ah-ha, so it IS bad".  I walked over to reboot it, and
> found I could actually log in on the console. check out this dmesg..
> 
> [  503.683055] Clocksource tsc unstable (delta = -95946009388 ns)
> [  503.692038] Switched to clocksource hpet
> [  524.420897] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! 
> [trinity-c178:20182]

Neat.  We often see switching to hpet on boxes as they are diving into 
softlockup pain, but it's not usually before the softlockups.

Are you configured for CONFIG_NOHZ_FULL?

I'd love to blame the only commit to kernel/smp.c between 3.16 and 3.17

commit 478850160636c4f0b2558451df0e42f8c5a10939
Author: Frederic Weisbecker <fweisbec@gmail.com>
Date:   Thu May 8 01:37:48 2014 +0200

    irq_work: Implement remote queueing

You've also mentioned a few times where messages stopped hitting the 
console?


commit 5874af2003b1aaaa053128d655710140e3187226
Author: Jan Kara <jack@suse.cz>
Date:   Wed Aug 6 16:09:10 2014 -0700

    printk: enable interrupts before calling 
console_trylock_for_printk()

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/