Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751903AbaDAG73 (ORCPT ); Tue, 1 Apr 2014 02:59:29 -0400 Received: from mail7.hitachi.co.jp ([133.145.228.42]:53398 "EHLO mail7.hitachi.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751153AbaDAG72 (ORCPT ); Tue, 1 Apr 2014 02:59:28 -0400 Message-ID: <533A63C8.70800@hitachi.com> Date: Tue, 01 Apr 2014 15:59:20 +0900 From: Masami Hiramatsu Organization: Hitachi, Ltd., Japan User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20120614 Thunderbird/13.0.1 MIME-Version: 1.0 To: Jovi Zhangwei Cc: Ingo Molnar , Steven Rostedt , LKML , Greg Kroah-Hartman , Frederic Weisbecker Subject: Re: Re: [PATCH 14/28] ktap: add runtime/kp_events.[c|h] References: <1396014469-5937-1-git-send-email-jovi.zhangwei@gmail.com> <1396014469-5937-15-git-send-email-jovi.zhangwei@gmail.com> <533930EF.8080703@hitachi.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org (2014/03/31 19:14), Jovi Zhangwei wrote: > On Mon, Mar 31, 2014 at 5:10 PM, Masami Hiramatsu > wrote: >> (2014/03/28 22:47), Jovi Zhangwei wrote: >>> kp_events.c handle ktap events management(registry, destroy, event callback) >>> >>> This file is core event management interface between ktap and kernel. >>> >>> Exposed functions: >>> 1). kp_events_init/kp_events_exit >>> >>> 2). kp_event_create_kprobe >>> create kprobe event, for example: >>> kdebug.kprobe("SyS_futex", function () {}) >>> >>> 3). kp_event_create_tracepoint >>> create tracepoint event, for example" >>> kdebug.tracepoint("sys_futex_enter", function () {}) >>> >>> 4). kp_event_create >>> create perf backend event, for example: >>> trace sched:sched_switch { print(argstr) } >>> >>> It call kernel function 'perf_event_create_kernel_counter' to >>> register event(tracepoint/kprobe/uprobe) >>> >>> 5). kp_event_getarg >>> get argument of event, from arg0 to arg9, >>> only can be called in probe context. >>> trace sched:sched_switch { print(arg0, arg1) } >>> >>> 6). kp_event_stringify/kp_event_tostr >>> stringify argstr, sometimes if store argstr as key to table, >>> then it need to stringify firstly, like below: >>> var s={} trace sched:sched_switch { s[argstr] += 1 } >>> (This is quite rare usage, but ktap support it) >>> >>> Note: >>> Why ktap support 'kdebug.kprobe' and 'kdebug.tracepoint' when >>> it already support perf backend event(trace xxx {})? >>> >>> Because benchmark shows raw kprobe and tracpoint interface is faster >>> than perf backed tracing, nearly 10+%, it's more fair to compare >>> with Systemtap by raw tracing syntax, not perf backend tracing. >>> >> >> Do we really need it just for a +10% performance? I doubt that. >> I think the benefit point of ktap is "dynamic & simple programmable >> tracer in kernel", not the good performance at least at this point. >> Thus I think we should start ktap only with perf backend. >> > Yeah, agreed, most people like the perf-backed tracing syntax, > that raw trace interface is just for benchmark when I wanted to look > overhead compare with stap, the result is very inspiring, ktap table > operation overhead is lower than stap. > > On the performance overhead of dynamic tracing tools(ktap/stap/dtrace), > it's interesting enough that dtrace was used in production many year, > _but_ IMO the runtime of dtrace is slow after I checked dtrace source > code :), system workload does big matter than tracing tool overhead. Yeah, I see that less overhead is also required especially for enterprise people. I just doubt that it is solved by ktap itself. Should we improve perf(or ftrace) to export more effective interfaces for this kind of tracers? Thank you, -- Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt@hitachi.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/