Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753726AbbGWBzW (ORCPT ); Wed, 22 Jul 2015 21:55:22 -0400 Received: from mga03.intel.com ([134.134.136.65]:55086 "EHLO mga03.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753638AbbGWBzV (ORCPT ); Wed, 22 Jul 2015 21:55:21 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.15,527,1432623600"; d="scan'208";a="611192944" Message-ID: <1437616519.22524.117.camel@tzanussi-mobl.amr.corp.intel.com> Subject: Re: [PATCH v9 00/22] tracing: 'hist' triggers From: Tom Zanussi To: Brendan Gregg Cc: Steven Rostedt , daniel.wagner@bmw-carit.de, Masami Hiramatsu , Namhyung Kim , josh@joshtriplett.org, Andi Kleen , LKML Date: Wed, 22 Jul 2015 20:55:19 -0500 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.10.4 (3.10.4-4.fc20) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2885 Lines: 82 Hi Brendan, On Wed, 2015-07-22 at 11:29 -0700, Brendan Gregg wrote: > G'Day Tom, > > On Thu, Jul 16, 2015 at 10:22 AM, Tom Zanussi > wrote: > > > > This is v9 of the 'hist triggers' patchset. > > > [...] > > I've browsed the functionality (sorry, catching up), and it looks like > this will solve a number of common problems. But it seems > tantalizingly close to solving a few more. These may already be on > your future todo list. > > A) CPU stack profiling > > Kernel stacktrace as a key will be hugely useful; is there a way to > enable this for a sampling profile? (eg, what perf record -F 99 does). > I take CPU profiles daily, and would prefer to aggregate stacks > in-kernel. Also, would like user stacktrace as a key (even if it's > just the hex). > This wasn't on my todo list but I can see how it would be useful. On the list now. ;-) > B) Key buckets > > Eg, imagine: > > echo 'hist:keys=common_pid.execname,count.log2:val=count' > > /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger > > to get a log2 bucketized histogram of syscall read request size. Same > for any value where using the value as a key gets too verbose, and you > just want a rough look at the distribution. (Would make it more > readable if it could also be sorted by the log2 value.) > This I actually had an early implementation of, which I plan on reviving... > C) Latency as a bucket key > > With kprobes, we could then have a log2 histogram of any function call > latency, collected efficiently. (There's already the function timers > in ftrace, which I'm using from function_graph with filters sets to > only match the target function.) > My original thought for doing this kind of thing was to generalize the concept of a 'difference'. I used it in v1 as a way to calculate the difference between requested and allocated sizes for memory allocations, which was kind of pointless, though convenient. The real value would of course be in applying it to inter-event values rather than intra-event. In this case it would be an inter-event difference between timestamps. And in my previous patchset, I had a 'function_hist' tracer, similar to and in fact based on the same code as function_graph, that would simply aggregate hitcounts for every function call in the kernel, which indeed was pretty efficient. So the pieces are or have been there to do something like this, just a matter of putting them together. > ... Those are the other common use cases, that the hist functionality > seemed suited for. Beyond that gets more custom, and we can use eBPF. > Exactly. Tom > Brendan -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/