Date: Fri, 19 Dec 2014 09:30:37 -0500
From: Chris Mason <clm@fb.com>
Subject: Re: frequent lockups in 3.18rc4
To: Dave Jones <davej@redhat.com>
CC: Linus Torvalds <torvalds@linux-foundation.org>,
        Mike Galbraith <umgwanakikbuti@gmail.com>,
        Ingo Molnar <mingo@kernel.org>, Peter Zijlstra <peterz@infradead.org>,
        =?iso-8859-1?q?D=E2niel?= Fraga <fragabr@gmail.com>,
        Sasha Levin <sasha.levin@oracle.com>,
        "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Suresh Siddha <sbsiddha@gmail.com>, Oleg Nesterov <oleg@redhat.com>,
        Peter Anvin <hpa@linux.intel.com>
Message-ID: <1418999437.13012.1@mail.thefacebook.com>
In-Reply-To: <20141219035859.GA20022@redhat.com>
References: <CA+55aFyUtZobUADgtss7e4w-yriMtz7hKVKs=Ed72KQoQ9n2nA@mail.gmail.com>
	<CA+55aFwE3gY+ChZtBpPtt_eY9nCj6pgF_wd8utRN9cOgRe2xOQ@mail.gmail.com>
	<20141215055707.GA26225@redhat.com>
	<CA+55aFyYYPgBfRdqtvqxdo==D_BOFSW3fvNULuVxPAr2vnq7Rg@mail.gmail.com>
	<CA+55aFz6KEKdz4Nxkx2fa-FH3PZ+Aa51iA_nXobsQ-dDW5PGEg@mail.gmail.com>
	<20141218051327.GA31988@redhat.com>
	<1418918059.17358.6@mail.thefacebook.com>
	<20141218161230.GA6042@redhat.com> <20141219024549.GB1671@redhat.com>
	<CA+55aFw7R+P+2xgXwUqG5uxNs=dLD59fL+6uHuhWHFtsAb1nSg@mail.gmail.com>
	<20141219035859.GA20022@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"; format=flowed
Sender: linux-kernel-owner@vger.kernel.org


On Thu, Dec 18, 2014 at 10:58 PM, Dave Jones <davej@redhat.com> wrote:
> On Thu, Dec 18, 2014 at 07:49:41PM -0800, Linus Torvalds wrote:
> 
>  > And when spinlocks start getting  contention, *nested* spinlocks
>  > really really hurt. And you've got all the spinlock debugging on 
> etc,
>  > don't you?
> 
> Yeah, though remember this seems to have for some reason gotten worse
> in more recent builds. I've been running kitchen-sink debug kernels
> for my trinity runs for the last three years, and it's only this
> last few months that this has got to be enough of a problem that I'm
> not seeing the more interesting bugs. (Or perhaps we're just getting
> better at fixing them in -next now, so my runs are lasting longer..)

I think we're also adding more and more debugging.  It's definitely a 
good thing, but I think a lot of them are expected to stay off until 
you're trying to track down a specific problem.  I do always run with 
CONFIG_DEBUG_PAGEALLOC here and lock debugging/lockdep, and aside from 
being slow haven't hit trouble.

I know it's 3.16 instead of 3.17, but 16K stacks are probably 
increasing the pressure on everything in these runs.  It's my favorite 
kernel feature this year, but it's likely to make trinity hurt more on 
memory constrained boxes.

Your trace with hrtimer debugging yesterday made some sense, but it 
still should have been survivable.  I mean you should have kept seeing 
lockups from that one poor task being starved out of filling up his 
pool.  I know you have traces with a ton more output, but I'm still 
wondering if usb-serial and printk from NMI really get along well.  I'd 
try with debugging back on and serial consoles off.  We carry patches 
to make oom print less, just because the time spent on our slow 
emulated serial console is enough to back the box up into a death 
spiral.

The fairness of spinlock debugging is a really great point too, 
definitely worth trying with that off (and fixing, I love spinlock 
debugging).

-chris


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/