The SystemTap team announces release 2.5, "boot loot"!
UEFI/SecureBoot support, boot-time probing, function callee probing,
improved error messages, better statement probing, improved function
prologue endings detection, and more!
= Where to get it
http://sourceware.org/systemtap/ - our project page
http://sourceware.org/systemtap/ftp/releases/systemtap-2.5.tar.gz
http://koji.fedoraproject.org/koji/packageinfo?packageID=615
git tag release-2.5 (commit 8f0fcd9)
There have been over 592 commits since the last release.
There have been over 74 bugs fixed / features added since the last
release.
= How to build it
See the README and NEWS files at
http://sourceware.org/git/?p=systemtap.git;a=tree
Further information at http://sourceware.org/systemtap/wiki/
= SystemTap frontend (stap) changes
- SystemTap now supports UEFI/SecureBoot systems, via
machine-owner-keys maintained by a trusted stap-server on the
network. (Key enrollment requires a one-time reboot and BIOS
conversation.)
- SystemTap is now capable of inserting modules early during the boot
process on dracut-based systems. See the 'onboot' command in
systemtap(8) for more information.
- DWARF probes can now use the '.callee[s]' variants, which allow more
precise function probing. For example, the probe point
process("myproc").function("foo").callee("bar")
will fire upon entering bar() from foo(). A '.callees' probe will
instead place probes on all callees of foo().
Note that this also means that probe point wildcards should be used
with more care. For example, use signal.*.return rather than
signal.*.*, which would also match '.callees'. See stapprobes(3stap)
for more info. This feature requires at least GCC 4.7.
- New switches have been added to help inspect the contents of installed
library files:
stap --dump-functions --> list all library functions and their args
stap --dump-probe-aliases --> list all library probe aliases
- Statement probes (e.g. process.statement) are now faster to resolve,
more precise, and work better with inlined functions.
- The heuristic algorithms used to search for function-prologue
endings were improved, to cover more optimization (or
lack-of-optimization, or incorrect-debuginfo) cases. These
heuristics are necessary to find $context parameters for some
function-call/entry probes. We recommend programs be built with
CFLAGS+=-grecord-gcc-switches to feed information to the heuristics.
- The stap --use-server option now more correctly supports address:port
type parametrization, for manual use in the absence of avahi.
- SystemTap now falls back on the symbol table for probing
functions in processes if the debuginfo is not available.
- SystemTap now reports more accurate and succinct errors on type
mismatches.
- The documentation for the SystemTap initscript service and the
SystemTap compile-server service have been completely converted from
README files to man pages (see systemtap(8) and stap-server(8)).
- Suggestions are now provided when markers could not be resolved. For
example, process("stap").mark("benchmart") will suggest 'benchmark'.
- SystemTap colors can now be turned off by simply setting
SYSTEMTAP_COLORS to be empty, rather than having to make it invalid.
= SystemTap script language changes
- SystemTap now supports a %( guru_mode == 0 /* or 1 */ %)
conditional for making dual-use scripts.
- Embedded-C functions may use STAP_RETURN(value) instead of the
more wordy STAP_RETVALUE assignment followed by a "goto out".
The macro supports numeric or string values as appropriate.
- The [s]println() function can now be called without any arguments to
simply print a newline.
= SystemTap runtime changes
- Fixed multiple potential kernel crashes by improving utrace and the
taskfinder.
- SystemTap now supports backtracing through its own, invoking module.
= SystemTap tapset changes
argv.stp support both translate-time and runtime args
arm64/registers.stp provide register access on ARM64
indent.stp new indent_depth() and thread_indent_depth()
functions to track indentation level
java.stp new [s]print_java_backtrace() functions to
retrieve backtrace in Java probes
aux_syscalls.stp new _stp_syscall_nr() function to retrieve
syscall number
many _*_str() functions have been simplified in
the way they translate enum values to strings
context.stp better CONFIG_USER_NS support
nfsd.stp ditto
task.stp ditto + new task_rlimit() function to retrieve
resource limits on a specific task
ioblock.stp better support for different kernels using
@choose_defined()
ip.stp ditto
loadavg.stp new tapset to retrieve and print load averages
in the usual 1-, 5-, and 15-minute span
rlimit.stp new tapset with function rlimit_from_str() to
retrieve the resource limit code from its string
scsi.stp probe scsi.iodone can alias to either a
tracepoint or module function
signal.stp tolerate NULL from pid2task() when retrieving
pid_name
syscalls.stpm new macros to help sys
task_ancestry.stp new task_ancestry() function to print the
parentage of a process
task_time.stp new task_start_time() retrieves the start time
of a specific task
new nsecs_to_string() to convert nanoseconds to
human-readable string
pn.stp new pnlabel() function to retrieve the name of
the fired label in a label probe
uconversions.stp new user_ulong() and user_ulong_warn() to
retrieve an unsigned long from user-space
[nd_]syscalls[2].stp split syscall.fork into syscall.fork,
syscall.vfork, and syscall.clone
better support for s390x
support for preadv/pwritev syscalls
- The types of the parameters in many syscall probe aliases are now more
accurate (located in [nd_]syscalls[2].stp)
- The argv tapset now merges translate-time and run-time positional
arguments, so all of these work:
stap -e 'probe oneshot {println(argv[1]," ",argv[2])}' hello world
stap -e 'probe oneshot {println(argv[1]," ",argv[2])}' \
-G argv_1=hello -G argv_2=world
staprun hello.ko argv_1=hello argv_2=world
- A new probe alias "oneshot" allows a single quick script fragment to run,
then exit.
- Some struct-sockaddr fields are now individually decoded for
socket-related syscalls:
probe syscall.connect { println (uaddr_af, ":", uaddr_ip) }
- The kprocess.exec probe has been updated to use syscall.execve, which
allows access to the new process' arguments (through the new 'argstr'
or 'args' variables) as well as giving better support across kernel
versions. Note also that the 'filename' variable now holds the
filename (quoted), or the address (unquoted) if it couldn't be
retrieved.
- The following tapset variables and functions are now deprecated:
- The 'clone_flags', 'stack_start', 'stack_size',
'parent_tid_uaddr', and 'child_tid_uaddr' variables in the
'syscall.fork' and 'nd_syscall.fork' probe aliases.
- The '_sendflags_str()' and '_recvflags_str()' functions have been
deprecated in favor of the new '_msg_flags_str()' function.
- The 'flags' and 'flags_str' variables in the 'syscall.accept' and
'nd_syscall.accept' probe alias.
= SystemTap sample scripts
- New samples:
python2.stp tapset example for python2
python3.stp tapset example for python3
pyexample.stp uses python tapset to print backtraces and
variable values
connect_stat.stp prints a task's ancestry whenever it attempts
to connect() to an IP address
nfsd-recent.stp tracks all nfsd server operations and reports
clients who made recent requests
procmod_watcher.stp monitors all calls to fork(), exec(), exit(),
init_module(), and delete_module() and prints
their info
pstree.stp generates a process diagram in DOT form (e.g.
can be useful to monitor a 'make' command)
rlimit_nofile.stp watches for processes that fail to allocate a
file descriptor
2048.stp play the classic game like never before, all
in kernel space!
- Changed samples:
interrupts-by-dev.stp print column headers once
last_100_frees.stp guard against 32-bit libc on 64 systems
mutex-contention.stp ditto
pfiles.stp support CONFIG_USER_NS
schedtimes.stp general cleanup and support for newer kernels
= Examples of tested kernel versions
2.6.18 (RHEL5 x86)
2.6.18 (RHEL5 x86_64)
2.6.32 (RHEL6 x86)
2.6.32 (RHEL6 x86_64)
3.10.0 (RHEL7 x86_64)
3.13.0 (Ubuntu 14.04 x86_64)
3.13.9 (Fedora 19 x86_64)
3.15-rc2 (Fedora Rawhide x86_64)
= Known issues with this release
- Some kernel crashes continue to be reported when a script probes
broad kernel function wildcards (PR2725). Issues have also been
reported with a possible relationship to the CONFIG_KPROBES_ON_FTRACE
code (RHBZ1093152).
- 32-on-64 bit userspace unwinding is truncated on older kernels, such
as 2.6.32 (PR15757)
- The dyninst backend is still very much a prototype, with a number
of issues, limitations, and general teething woes. For instance:
+ lack of support for multiarch/cross-instrumentation
+ tapset functions are still incomplete relative to what is supported
when the kernel backend is active
+ exception handling becomes completely broken in programs
instrumented by the current version of dyninst (PR14702)
+ command line interrupts are slightly mishandled (PR15049)
+ not all registers are made available on 32-bit x86 (PR15136)
See dyninst/README and the systemtap/dyninst Bugzilla component
(http://tinyurl.com/stapdyn-PR-list) if you want all the gory
details about the state of the feature.
= Contributors for this release
Aaron Tomlin*, Brent Baude*, Dave Brolley, David Juran*, David Smith,
Fahad Arshad*, Frank Ch. Eigler, Jonathan Lebon, Josh Stone, Lukas
Berk, Mark Wielaard, Martin Cermak, Masatake YAMATO*, Miroslav Franc,
Negreanu Marius Adrian, Peter Kjellström*, Quentin Barnes*, Robin
Hack*, Stan Cox, Tomoki Sekiyama, Torsten Polle, Victor Kamensky*,
William Cohen
Special thanks to new contributors, marked with '*' above.
Special thanks to Jonathan Lebon for compiling these notes.
= Bugs fixed for this release <http://sourceware.org/PR#####>
1133 support .callees probe pattern extension
2035 investigate boot-time probing
5154 aux_syscalls lookup tables needlessly included
6941 better prologue heuristics needed for userspace (non-regparm)
10208 Support probing glibc synthesized syscall wrappers
10300 Provide more macros for embedded-C functions
11113 utrace changes in 2.6.32 kernel cause task_finder.c/itrace.c compile errors
13420 prologue detection fails for function parameters in unoptimized (-g only, no -O) code
14223 perf probes fail if not root
14436 warn about use of $vars (e.g. in return probes) with unoptimized object code with poor VTA
15219 syscall.exp failures on RHEL5, RHEL6, and rawhide
15781 improve pass-2 function suggestions
15807 need more serious workload program for examples vice -c sleep 0.2
15907 listing probe lines gives odd results
15999 symbol-table-based fallback for foo.function() should resolve $$vars etc.
16097 improve error message on array type mismatch
16129 intermittent RCU/sleep warnings from perf probes / utrace_death handlers
16132 staprun/stapio confused by presence of traceN files
16162 plt addresses are incorrect for prelinked libraries
16165 @perf operator in error msgs is corrupted
16166 stap -vvv causes SIGSEGV when using @perf operator
16175 on ppc64, getting mmap syscall and nd_syscall testsuite failures
16176 on ia32, the trunc nd_syscall.exp subtest fails
16184 The rt_sigprocmask syscall subtest fails
16187 the 'sendfile' syscall.exp/nd_syscall.exp test fails
16207 on rawhide, we're getting more syscall.exp failures
16247 Some examples don't work on RHEL5
16248 pfiles example fails to build on RHEL7
16276 resource limits interfering with buildrun.cxx make -j
16278 orphaned stapsh goes to 100% CPU after stap closing
16307 ubuntu 12.04 /bin/kill no likey new kill -SIG -- -[exp_pid]
16309 SELinux prevents stap-serverd from opening the current dir when spawning processes
16311 need to add pwritev() syscall support to the syscall/nd_syscall tapsets
16312 need to add preadv() syscall support to the syscall/nd_syscall tapsets
16319 on sysvinit systems, virtio-serial port hotplugging fails
16326 client.exp and local stap-server interactions
16342 stap --list-servers lists spurious line when having two stap-serverd instances
16378 in-kernel utrace problem with 2.6.32: implicit declaration of function 'do_each_thread' in task_finder.c
16406 pass-5 build-id mismatch when -c cmd has same name as module
16448 adapt stap-prep for el7
16460 on ia64, the syscall.unlink probe alias has compile errors
16471 the server_concurrency.exp testcase has problems
16472 long chain of nested function calls consumes a lot of empty whitespace due to thread_indent() function
16473 rlimit.exp: adjust for weak symbols
16476 netfilter API changed for linux 3.13
16478 stap -l process.function.label listing not just labels
16514 String comparation doesn't work in kprocess.exec probe.
16540 the scsi tapset could use the scsi tracepoints
16541 nfsd tapset has errors
16542 the rpc tapset has errors
16555 on rawhide, we're getting errors in the task.stp tapset
16557 on RHEL5, the vm.mmap probe alias needs help
16570 the syscall.exp and nd_syscall.exp testcases take too long
16580 syscall.socket and syscall.socketpair should decode the 'protocol' argument
16603 time loss-of-synchrony on fluctuating-cpufreq unsynchronized-tsc boxes
16615 don't require access to dwarf_query in has_single_line_record()
16616 the 'servername' convenience variable returned by the rpc tapset can be NULL
16633 odd 'stap -L' behavior
16636 pwritev/preadv syscall testsuite bug on RHEL5
16659 On kernels >= 3.13, the __ip_sock_daddr() tapset function no longer compiles
16660 On kernels >= 3.14, the ioblock.request 'sector' variable is broken
16665 [PATCH] - DRAFT - Add new functions for resource limits investigation of task_struct.
16666 Symtab functionality needs a strip -g testcase
16667 need to add clone() syscall support to the syscall/nd_syscall tapsets
16676 Inconsistently-biased addresses for ET_EXEC
16697 inode-uprobes cleanup race
16718 kernel crash in _stp_copy_from_user()
16719 systemtap runs hold vmlinux file descriptor open
16725 stap should warn when system wide process probes are used with --runtime=dyninst
16726 RFE: provide a way to retrieve tapset function types
16730 stap -L '**' displays non-public probe points
16766 possible kernel crash after failed initialization & flight-recorder mode
16806 kernel crash during repeated module insertion
16829 Trigger STAPBM_VERBOSE=true automatically when -v's are specified with java probes
16884 [nd_]syscall.shmctl and [nd_]syscall.compat_shmctl provide different variables