On Wed, Oct 13, 2010 at 4:50 PM, Steven Rostedt <[email protected]> wrote:
> On Wed, 2010-10-13 at 16:19 -0700, David Sharp wrote:
>> Google uses kernel tracing aggressively in the its data centers. We
>
> Thanks!
>
>> wrote our own kernel tracer, ktrace. However ftrace, perf and LTTng
>> all have a better feature set than ktrace, so we are abandoning that
>> code.
>
> Cool!
>
>>
>> We see several implementations of tracing aimed at the mainline kernel
>> and wanted a fair comparison of each of them to make sure they will
>> not significantly impact performance. A tracing toolkit that is too
>> expensive is not usable in our environment.
>>
>
> [ snip for now (I'm traveling) ]
>
>> This first set of benchmark results compares ftrace to ktrace. The
>> numbers below are the "on" result minus the "off" result for each
>> configuration.
>>
>> ktrace: 200ns (tracepoint: kernel_getuid)
>> ftrace: 224ns (tracepoint: timer:sys_getuid)
>> ftrace: 587ns (tracepoint: syscalls:sys_enter_getuid)
>
>
>> The last result shows that the syscall tracing is about twice as
>> expensive as a normal tracepoint, which is interesting.
>
> Argh, the syscall tracing has a lot of overhead. There is only one
> tracepoint that is hooked into the ptrace code, and will save all
> registers before calling the functions. It enables tracing on all
> syscalls and there's a table that decides whether or not to trace the
> syscall.
>
> So I'm not surprised with the result that the syscall trace point is so
> slow (note, perf uses the same infrastructure).
>
> -- Steve
Steve, Mike told me that you may have some patches to improve overhead
of syscall tracing. I'd be interested in testing them. Can you send
them my way?
Thanks,
d#
On Tue, 2010-11-16 at 15:27 -0800, David Sharp wrote:
> Steve, Mike told me that you may have some patches to improve overhead
> of syscall tracing. I'd be interested in testing them. Can you send
> them my way?
Mike may have been confused ;-)
I did tell him that some ring buffer improvements did make it into the
kernel. You might want to try the latest mainline again.
I even have other patches that might help. I'll have to go and dig them
up.
-- Steve
On Tue, Nov 16, 2010 at 3:32 PM, Steven Rostedt <[email protected]> wrote:
> Mike may have been confused ;-)
I am always confused.
> I did tell him that some ring buffer improvements did make it into the
> kernel. You might want to try the latest mainline again.
You told me that you had patches (in 2.6.37) I think that would allow
us to specifically improve the syscall tracing performance.
Can you point us at em?
mrubin
On Thu, 2010-11-18 at 17:11 -0800, Michael Rubin wrote:
> > I did tell him that some ring buffer improvements did make it into the
> > kernel. You might want to try the latest mainline again.
>
> You told me that you had patches (in 2.6.37) I think that would allow
> us to specifically improve the syscall tracing performance.
>
> Can you point us at em?
Look at commits:
afcc5c6872f0215d515a637041bb51f8691a8ea7
69d1b839f7eee347e357b3f6cce7f630cc6ff93d
140ff89127c74b1b1c1b0152a36ea3720ccf6bc3
d9abde2138e0a00a0d7e44676928efa0ef629d48
b8b2663bd7c9da04ac804659b9f617c199d0252c
-- Steve