Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751629Ab3JYV21 (ORCPT ); Fri, 25 Oct 2013 17:28:27 -0400 Received: from mail-qa0-f43.google.com ([209.85.216.43]:46589 "EHLO mail-qa0-f43.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751265Ab3JYV20 (ORCPT ); Fri, 25 Oct 2013 17:28:26 -0400 Date: Fri, 25 Oct 2013 17:36:15 -0400 (EDT) From: Vince Weaver To: linux-kernel@vger.kernel.org cc: Steven Rostedt , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Dave Jones , Frederic Weisbecker Subject: Re: perf/ftrace lockup on 3.12-rc6 with trigger code In-Reply-To: Message-ID: References: <1382695687.12254.4.camel@pippen.local.home> User-Agent: Alpine 2.10 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1664 Lines: 47 On Fri, 25 Oct 2013, Vince Weaver wrote: > > I'm not sure how tracepoints work exactly, but the problem code is setting > pe[5].type=PERF_TYPE_TRACEPOINT; > pe[5].config=0x7fffffff00000001; > > The config is being truncated to 32-bits by the perf/ftrace code so I > think this means the tracepoint being enabled is > > tracing/events/ftrace/function/id:1 > I've wasted much of the day playing with this and adding printks, etc. The key things that cause the problem are: tracepoint event config is 1 (tracing/events/ftrace/function) PERF_SAMPLE_PERIOD set in the sample type no user mmap buffer period (not frequency) enabled What this means is there is pretty high number of kernel calls happening. Every kernel function entry ends up calling the perf_swevent_overflow() overflow handler, which calls perf_event_output() which attempts to dump the buffer, but it can't because no user buffer is mmap'd. This causes some sort of storm and eventually the system just stops responding and the watchdog kicks in, although the traces it gives back are different each time. It's possible the kernel is making forward progress (though very slowly) and this is just some sort of throttling issue. I don't know if there are any better ways to try to debug things than the printk route. Though that has its own problems as the printk's themselves start showing up in the ftrace traces. Vince -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/