The SystemTap team announces release 4.0!
prometheus exporter network service; ebpf support extensions including
strings and implementation of traditional log(), sprintf() functions;
rebuilt rich tapset coverage for 4.17+ syscalls and for
tracepoint-based syscalls; script language tweaks for supporting
machine-generated scripts
= Where to get it
https://sourceware.org/systemtap/ - our project page
https://sourceware.org/systemtap/ftp/releases/
https://koji.fedoraproject.org/koji/packageinfo?packageID=615
git tag release-4.0 (commit 428f84e9e656b)
There have been over 250 commits since the last release.
There have dozens of bugs fixed / features added since the last release.
= SystemTap frontend (stap) changes
- A new network service, stap-exporter, is included. It glues
systemtap and the web. It allows a prometheus (or compatible
systems such as pcp) to consume metrics exported by systemtap
scripts. Some tapset macros/functions are available to make it
easier to write such scripts. See the stap-exporter(8) man page and
the systemd service.
- Embedded-C functions marked /* guru */ may now be invoked from other
tapset probes / functions, while still being invalid for normal call
from an unprivileged user script.
- New script language operators @kderef/@uderef and @kregister/@uregister
were added.
@kderef/@uderef (size,address) can be used to dereference integers and
@kregister/@uregister (dwarf#) can be used to access register values.
- The eBPF backend's string support has been improved. Strings
can now be stored in variables, passed as function arguments,
and stored as array keys and values.
- The 3rd operand of the ternary operator '?:' in the script language
now binds tighter than the binary assignment operators like '=' and
'+=', just like the C language. The original operator precedence can
be restored by the '--compatible 3.3' option.
- The script language now supports the use of bare 'return' statements
(without any return values) inside functions which do not return any
values. A trailing semicolon is recommended for such return
statements to avoid any potential ambiguity. The parser treats a
following semicolon (';') or a closing curly bracket ('}') as a
terminator for such bare return statements.
- Parentheses after unary '&' with a target-symbol expression is
now accepted in the script language.
- A systemd service file has been added for systemtap.service (which
runs a configurable set of scripts automatically on system
startup). The existing /etc/init.d/systemtap init script has been
moved to a new utility command 'systemtap-service' which preserves
functionality such as configuring onboot systemtap scripts via
dracut. See systemtap-service(8) for details.
= SystemTap backend changes
- Runtime/tapsets were ported to include up to kernel version 4.19-rc
- When a systemtap module is loaded, the name of the original stap script
is now printed to dmesg by the kernel runtime.
- On some Fedora kernels, the information necessary to automatically
engage in SecureBoot module signing is hidden from systemtap.
Setting the $SYSTEMTAP_SIGN environment variable forces it on.
A running stap-server instance will also be needed.
- The experimental ftrace ring buffer mechanism (STP_USE_RING_BUFFER)
has been deprecated and may be removed in future versions.
= SystemTap tapset changes
- The syscall tapset is now updated to work on kernel 4.17+.
Additionally, the tapset now includes an automatic fallback alias to
the sys_enter / sys_exit kernel tracepoints, if no other
kprobe-based mechanism is found. These changes have brought
unavoidable consequences. Raw $target variables for the syscall
arguments and return probes (e.g. @entry($fd), $return, returnval())
may not longer be relied upon. Instead, use the variables defined by
the tapset aliases. For example:
% stap -L syscall.read
syscall.read name:string fd:long buf_uaddr:long count:long argstr:string
% stap -L syscall.read.return
syscall.read.return name:string retval:long retstr:string
to see the available variables for that syscall. See
[man stapprobes] for further details. returnval() in particular is
being deprecated soon; use retval in syscall.*.return probes instead.
- Tapset functions register() and u_register() now support 8-bit
x86 register names "ah", "al", "bh", "bl", "ch", "cl", "dh", and
"dl" on both x86_64 and i386. And 16-bit x86 registers are now
truly read as 16-bit integers instead of as 32-bit ones.
- Added a new abort() function that combines exit() and an immediate
return from the current probe/function.
= SystemTap sample scripts
All 180+ examples can be found at https://sourceware.org/systemtap/examples/
- New samples exporting data in Prometheus format:
also_ran.stp Keep a tally of executables run on the system
cpu_throttle.stp Monitor Intel processors for throttling
due to power or thermal limits
syscallsbypid.stp Provide a per-process syscall tally on the system
syscallerrorsbypid.stp Provide a per-process syscall error tally
syscalllatency.stp Provide a per-process accumulation of syscall latency
- New stap-exporter-scripts/ subdirectory in systemtap.examples.
- Numerous example script improvements and new samples galore:
gmalloc_watch.stp Tracing glib2 memory allocations
ioctl_handler.stp Monitor which executables use ioctl syscalls
and what kernel code is handling the ioctl
libguestfs_log.stp Trace libguestfs startup
measureinterval.stp Measure intervals between events
php-trace.stp Tracing of PHP code execution
stap_time.stp Provide elapsed times for passes
of SystemTap script compilation
tcl-funtop.stp Profile Tcl calls
tcl-trace.stp Callgraph tracing of Tcl code
cve-2018-14634.stp historical emergency security band-aid,
for reference/education only
= Examples of tested kernel versions
2.6.32 (RHEL 6 x86_64, i686)
3.10.0 (RHEL 7 x86_64)
4.15.0 (Ubuntu 18.04 x86_64)
4.16.13 (Fedora 28 x86_64)
4.18.0 (Fedora x86_64)
4.18.12 (Fedora 28 x86_64, arm64, ppc64)
4.19-rc7 (Fedora Rawhide x86_64)
= Known issues with this release
- Some kernel crashes continue to be reported when a script probes
broad kernel function wildcards. (PR2725)
- An upstream kernel commit #2062afb4f804a put "-fno-var-tracking-assignments"
into KCFLAGS, dramatically reducing debuginfo quality, which can cause
debuginfo failures. The simplest fix is to erase, excise, nay, eradicate
this line from the top level linux Makefile:
KBUILD_CFLAGS += $(call cc-option, -fno-var-tracking-assignments)
= Coming soon
- prometheus-exporter is here, more tasty systemtap & http chocolate en route
= Contributors for this release
Aaron Merey, David Smith, Frank Ch. Eigler, Jafeer Uddin, Martin Cermak,
Masanari Iida, *Paulo Andrade, Serhei Makarov, Stan Cox, Victor Kamensky,
William Cohen, Yichun Zhang (agentzh), *Zexuan Luo
Special thanks to new contributors, marked with '*' above.
Special thanks to Serhei Makarov for assembling these notes.
= Bugs fixed for this release <https://sourceware.org/PR#####>
14690 the syscall tapsets could be written to prefer the 'syscalls' tracepoints
21888 bpf variants of log()/etc. functions
22310 build parser syntax for all the new staptree types
23160 4.17 breaks syscalls tapset
23284 dmesg should identify the name of the stap script
23356 server.exp test case hangs on rawhide
23359 impose security constraints on @kderef, @kregister
23407 bpf: backend should support strings as first class values
23480 bpfinterp.cxx should respond to ^C
23488 support CONFIG_DEBUG_INFO_REDUCED builds
23510 Tapset function println() not supported in the bpf runtime
23599 Use of usymname() with stap -u leads to kernel module compilation errors
23608 long stapregex overflows arc_priority
23666 Aggregate operations specified in foreach loop is not respected by the translator
23736 rawhide 4.19 kernel panic during tracepoint enumeration
23760 .statement() wildcard probes fail if any cu/srcfile lacks debug_line data
23766 staprun -R (default) fails for modules with short hardcoded -m names