Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752481AbbGVSpj (ORCPT ); Wed, 22 Jul 2015 14:45:39 -0400 Received: from mail-ig0-f173.google.com ([209.85.213.173]:37454 "EHLO mail-ig0-f173.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750779AbbGVSph (ORCPT ); Wed, 22 Jul 2015 14:45:37 -0400 MIME-Version: 1.0 In-Reply-To: References: From: Brendan Gregg Date: Wed, 22 Jul 2015 11:29:47 -0700 Message-ID: Subject: Re: [PATCH v9 00/22] tracing: 'hist' triggers To: Tom Zanussi Cc: Steven Rostedt , daniel.wagner@bmw-carit.de, Masami Hiramatsu , Namhyung Kim , josh@joshtriplett.org, Andi Kleen , LKML Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1764 Lines: 50 G'Day Tom, On Thu, Jul 16, 2015 at 10:22 AM, Tom Zanussi wrote: > > This is v9 of the 'hist triggers' patchset. > [...] I've browsed the functionality (sorry, catching up), and it looks like this will solve a number of common problems. But it seems tantalizingly close to solving a few more. These may already be on your future todo list. A) CPU stack profiling Kernel stacktrace as a key will be hugely useful; is there a way to enable this for a sampling profile? (eg, what perf record -F 99 does). I take CPU profiles daily, and would prefer to aggregate stacks in-kernel. Also, would like user stacktrace as a key (even if it's just the hex). B) Key buckets Eg, imagine: echo 'hist:keys=common_pid.execname,count.log2:val=count' > /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger to get a log2 bucketized histogram of syscall read request size. Same for any value where using the value as a key gets too verbose, and you just want a rough look at the distribution. (Would make it more readable if it could also be sorted by the log2 value.) C) Latency as a bucket key With kprobes, we could then have a log2 histogram of any function call latency, collected efficiently. (There's already the function timers in ftrace, which I'm using from function_graph with filters sets to only match the target function.) ... Those are the other common use cases, that the hist functionality seemed suited for. Beyond that gets more custom, and we can use eBPF. Brendan -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/