Hi,
I have reworked the underlying buffering mechanism LTTng uses in the
past week. I took "relay", changed its vmap() buffers for my own linked
list of buffers (not vmaped), made read/write wrappers which support
writing data larger than a page size and writing across page boundaries,
and finally managed to create a splice_read file operation which
supports that. I finally changed lttd to make it use splice() instead of
a mmap() and... it worked! :-) (after a bit a debugging, clearly)
This is all in LTTng 0.27 and ltt-control 0.53 (for the lttd part).
As this is an important change, testing is very welcome. If you are
interested in looking in the inner details of the buffering mechanism I
just did, you might also want to enable the "check for random buffer
access" option in the menuconfig. It will generate warnings when offset
accesses more than a page away from the previous done is requested by
the client. Cases such as large data write and reentrancy over the
tracer code will generate a few "cache misses", but it's supposed to be
rare, and therefore not a performance concern. This self-checking
feature has proven to be very useful in the early development stages,
and I think it's also useful if any other tracer client want to use it :
it helps finding lack of reference locality a tracer client might have.
I am also very interested in getting numbers comparing the performance
of the new buffering infrastructure with the previous one. Having much
less TLB impact should improve performance.
It's all available in the git tree :
http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=summary
And the userspace packages available at http://ltt.polymtl.ca (see the
"QUICKSTART")
Cheers,
Mathieu
--
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
I did some performance testing between LTTng 0.26 (uses vmap buffers)
and LTTng 0.31 (vmap-less buffers). I think performance got degraded in
the process. Those tests only write trace data in circular buffers
("overwrite mode").
No tracing, kernel 2.6.27-rc8 :
tbench : 1862.24
LTTng 0.26, kernel 2.6.27-rc7, flight recorder, default size buffers :
tbench : 1156.37 MB/s
LTTng 0.31, kernel 2.6.27-rc8, flight recorder, default size buffers :
tbench : 942.72 MB/s
For those of the LTT community interested in LTTng performance, I think
it's worth having a look at the ltt-relay-alloc.patch implementation to
see if there is any obvious performance degradation source. I'll play a
bit with the implementation to see where it comes from.
Mathieu
* Mathieu Desnoyers ([email protected]) wrote:
> Hi,
>
> I have reworked the underlying buffering mechanism LTTng uses in the
> past week. I took "relay", changed its vmap() buffers for my own linked
> list of buffers (not vmaped), made read/write wrappers which support
> writing data larger than a page size and writing across page boundaries,
> and finally managed to create a splice_read file operation which
> supports that. I finally changed lttd to make it use splice() instead of
> a mmap() and... it worked! :-) (after a bit a debugging, clearly)
>
> This is all in LTTng 0.27 and ltt-control 0.53 (for the lttd part).
> As this is an important change, testing is very welcome. If you are
> interested in looking in the inner details of the buffering mechanism I
> just did, you might also want to enable the "check for random buffer
> access" option in the menuconfig. It will generate warnings when offset
> accesses more than a page away from the previous done is requested by
> the client. Cases such as large data write and reentrancy over the
> tracer code will generate a few "cache misses", but it's supposed to be
> rare, and therefore not a performance concern. This self-checking
> feature has proven to be very useful in the early development stages,
> and I think it's also useful if any other tracer client want to use it :
> it helps finding lack of reference locality a tracer client might have.
>
> I am also very interested in getting numbers comparing the performance
> of the new buffering infrastructure with the previous one. Having much
> less TLB impact should improve performance.
>
> It's all available in the git tree :
> http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=summary
>
> And the userspace packages available at http://ltt.polymtl.ca (see the
> "QUICKSTART")
>
> Cheers,
>
> Mathieu
>
> --
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
>
> _______________________________________________
> ltt-dev mailing list
> [email protected]
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>
--
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68