Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753570AbcC1CKT (ORCPT ); Sun, 27 Mar 2016 22:10:19 -0400 Received: from mx1.redhat.com ([209.132.183.28]:40669 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752960AbcC1CKQ (ORCPT ); Sun, 27 Mar 2016 22:10:16 -0400 Date: Sun, 27 Mar 2016 22:10:13 -0400 From: "Frank Ch. Eigler" To: systemtap@sourceware.org Cc: linux-kernel@vger.kernel.org, lwn@lwn.net Subject: systemtap 3.0 release Message-ID: <20160328021013.GA22488@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 13239 Lines: 292 The SystemTap team announces release 3.0! Experimental monitor and interactive modes, optimized associative arrays, two types of function overloading, probe point brace-pattern expansion, security band-aid samples, improved string quoting and escaping, pretty printing array aggregates, private scoping for variables / functions, oxford commas, and much more! = Where to get it https://sourceware.org/systemtap/ - our project page https://sourceware.org/systemtap/ftp/releases/systemtap-3.0.tar.gz https://koji.fedoraproject.org/koji/packageinfo?packageID=615 git tag release-3.0 (commit 616ec7a0b916df) There have been over 450.32 commits since the last release. There have been between 57 and 60 bugs fixed / features added since the last release. = How to build it See the README and NEWS files at https://sourceware.org/git/?p=systemtap.git;a=tree Further information at https://sourceware.org/systemtap/wiki/ = SystemTap frontend (stap) changes - The new experimental "interactive" mode, specified by "stap -i", drops you into a command-line prompt where you can build up a script, run it, edit it, run it again, etc. Type "help" for a list of commands. - New experimental --monitor[=INTERVAL] option similar to unix "top". This allows users to see statistics about the running module(uptime, module name, invoker uid, memory sizes, global variables, and the current probe list along with their statistics). An interface is also provided to allow control over the running module (resetting global variables, sorting the list of probes, deactivating and reactivating probes). - Add macros @prints to print a scalar aggregate variable, @prints[1-9] to print an array aggregate (of given index-arity), formatted similarly to the automatic printing of written-only global variables. global a, b probe oneshot { a <<< 1; b[tid()] <<< 2 } probe end { @prints(a); @prints1(b) } - Add Czech version of key manual pages. - The stap compile server will log the stap client's options that are passed to the server. The options that get logged on the server will include the script name or the -e script, depending on which is used by the client. = SystemTap script language changes - The performance of associative arrays have been dramatically improved, especially for densely filled tables and for multiple indexes. The hash tables behind these arrays is now sized as a function of the array maximum size with an optional MAPHASHBIAS space/time tradeoff knob. - Functions may now be overloaded during module runtime using the "next" statement in script functions and STAP_NEXT macro for embedded-C functions. They may also be overloaded by number of parameters during compile time. For example, Runtime overloading: function f() { if (condition) next; print("first function") } function f() %{ STAP_NEXT; print("second function") %} function f() { print("third function") } For the given functions above, a functioncall f(), will execute the body of the third function if condition evaluates to true and print "third function". Note that the second function is unconditionally nexted. Parameter overloading: function g() { print("first function") } function g(x) { print("second function") } g() -> "first function" g(1) -> "second function" Note that runtime overloading does not occur in the above example as the number of parameters of the functions differ. The use of a next statement inside a function while no more overloads remain will trigger a runtime exception. The function candidates are selected at compile time and is determined by the number of arguments provided for the functioncall. - Embedded-C functions and blocks may now access script level global variables using the STAP_GLOBAL_GET_* and STAP_GLOBAL_SET_* macros. To read or write the script global var, the /* pragma:read:var */ or /* pragma:write:var */ marker must be placed in the embedded-C function or block. The written type must match the type inferred at script level. Scalars: STAP_GLOBAL_SET_var(STAP_GLOBAL_GET_var()+1) -> increments script global var by 1 STAP_GLOBAL_SET_var("hello") Associative arrays: STAP_GLOBAL_GET_var(index-1, ..., index-n) STAP_GLOBAL_SET_var(index-1, ..., index-n, new value) - Probe point brace expansion is now supported to improve brevity in specifying probe points. For example, process.{function("a"), function("b").{call,return}} => process.function("a"), process.function("b").call, process.function("b").return process.{function("*").{callees,plt}? => process.function("*").callees?, process.plt? {kernel,module("nfs")}.function("nfs*")! => kernel.function("nfs*")!, module("nfs").function("nfs*")! - Profiling timers at arbitrary frequencies are now provided and perf probes now support a frequency field as an alternative to sampling counts. probe timer.profile.freq.hz(N) probe perf.type(N).config(M).hz(X) The specified frequency is only accurate up to around 100hz. You may need to provide a higher value to achieve the desired rate. - Added support for private global variables and private functions. The 'private' keyword limits these to the tapset file they are defined in. = SystemTap tapset changes ansi.stp Functions ansi_set_color{2,3} are replaced by overloaded ansi_set_color linux/[arm/]aux_syscalls.stp Support for arm kernels less than 3.7. linux/arm/[nd_]syscalls.stp Support for [nd_]syscall.execve for arm kernels less than 3.7. linux/aux_syscalls.stp New _stp_mlock2_str function to convert mlock2 syscall flags to a string. linux/context.stp New module_size() function. linux/conversions.stp - New kernel_string_quoted_utf[16|32] functions combines @string_quoted and @kernel_string_utf* - kernel_string* functions with alternative error strings are replaced by overloaded variants linux/nd_syscalls.stp Add nd_syscall.mlock2 kprobe based probe point linux/perf.stp Update recent uapi/linux/perf_event.h bits linux/proc_mem.stp proc_mem_*_pid functions are replaced by overloaded proc_mem_* linux/syscalls.stp Add syscall.mlock2 kernel function probe point linux/task.stp New task_cwd_path and task_exe_file functions linux/task_time.stp task_{s,u}time_tid functions are replaced by overloaded task_{s,u}time linux/uconversions.stp user_string functions with alternative warning strings are replaced by overloaded variants logging.stp New overloaded variant of assert, assert(expression) print_stats.stpm @prints* macros for printing stats timers.stp Probe point timer.profile.freq for profiling try_assign.stpm New @try_assign macro uconversions.stp Add user_string_quoted_utf[16|32] function that quotes a given UTF-[16|32] string from a given user address - Internal tapset functions and global variables are marked as private, where possible. - Some tapsets have been modified to make use of the new function overloading feature. Instead of having new function names with suffixes such as "2" or "pid" to indicate extra arguments, the functions now seem to have optional arguments. - New tapset function string_quoted() to quote and \-escape general strings. String $context variables that are pretty-printed are now processed with such a quotation engine, falling back to a 0x%x (hex pointer) on errors - Functions get_mmap_args() and get_32mmap_args() got deprecated. = SystemTap sample scripts - now at 156 samples! who_sent_it.stp Trace outgoing network packets using the netfilter probes, printing the source thread name/id and destination host:port - New to collection is a selection of security band aids for specific CVEs. They are historical emergency security band-aids, and are for reference/education only. The scripts can be found under the security-band-aids folder in the examples directory. - A number of samples were tweaked for portability and demonstration of newer language/tapset facilities. = Examples of tested kernel versions 2.6.18 (RHEL 5 x86 and x86_64) 2.6.32 (RHEL 6 x86 and x86_64) 3.10.0 (RHEL 7 x86_64) 4.1.6 (Fedora 22 x86_64) 4.3.4 (Fedora 22 x86_64) 4.6.0-rc0 (Fedora rawhide x86_64) = Known issues with this release - Some kernel crashes continue to be reported when a script probes broad kernel function wildcards. (PR2725) - The dyninst backend is still very much a prototype, with a number of issues, limitations, and general teething woes. See dyninst/README and the systemtap/dyninst Bugzilla component ( http://tinyurl. com/stapdyn-PR-list ) if you want all the gory details about the state of the feature. - An upstream kernel commit #2062afb4f804a put "-fno-var-tracking-assignments" into KCFLAGS, reducing debuginfo quality which can cause debuginfo failures. A proposed workaround to this issue exists in: https://lkml.org/lkml/2014/11/21/505 . Fedora kernels are not affected by this issue. = Contributors for this release Abegail Jakop, David Smith, Felix Lu, Frank Ch. Eigler, Jesper Brouer*, Jonathan Lebon, Josh Stone, Ken Raeburn*, Lukas Berk, Mark Wielaard, Martin Cermak, Nan Xiao*, Nikolay Borisov*, Orivej Desh*, Stan Cox, Torsten Polle, William Cohen, Zhou Wenjian* Special thanks to new contributors, marked with '*' above. Special thanks to Abegail for compiling these notes. = Bugs fixed for this release 6525 need utrace task-finder-based pid->execname, pid->cwd-path-name tables 6977 Incorrect resolution of variables in function scope 9968 Share global variable between embedded C and script 10607 need way to protect sensitive tapset globals 11347 intra-tapset visibility scoping for embedded-c functions 12414 Missing return value is not reported 13514 more robust -rt support 13608 Limit the size of the request from the compile server client 13643 log stap options 13721 local variable name collision 13820 timer.profile(NNN) variant using perf probes 14203 Embedded-C functions in various tapsets are not stress tested 14431 char * always being printed as a, possibly "", without the address 15775 relay host/guest functionality is broken on RHEL6 16672 Allow brace expansion on probe points 17393 java.exp: ERROR: Cannot attach to module control channel 17587 too sensitive to duplicate globals across tapset files 17741 try-catch statement as a tapset function 18075 Add a return-probe variant of .callee 18431 runtime function overloading facility 18848 new --monitor option 18876 need to assert 18884 const-folded literals get unnecessary tmp storage 18889 lost ability to probe kernel module initializers 19120 kbuild warnings on RHEL5 19135 testsuite results hard to compare 19136 "private" keyword for tapsets 19216 kprobe.module("MOD").function("FOO") doesn't work on not loaded modules 19234 user_string_n2_quoted() vs. user_string_n_quoted.exp 19236 NULL global variable breakage 19241 systemtap fails to start when CPU0 is offline 19265 finding out-of-tree kernel modules works with full paths, not relative paths 19275 the stap_run testsuite function doesn't handle unexpected output 19345 RHEL 7.0 s390x crash in check.exp 19346 escape xml documentation 19358 netfilter probes are broken on rawhide 19362 translator puts two probes for each probe point in .../library.exp tests 19396 systemtap can't find certain kernel tracepoints 19470 process_by_pid.exp always fails on arches that don't support .plt probes 19497 linux 4.4 commit 7523e4dc50 19502 extend stap.1 with documentation overview 19510 the "private" keyword support has made -p1 output less useful 19521 the "private" keyword support has made error messages less useful 19525 script doesn't finish in bulk mode 19536 kretprobe data-pouch type issue 19537 the parseok/semko.stp test case always fails 19560 flight recorder mode intermittently fails 19639 parallel_exec.exp fails in beaker-project.org 19644 linux 4.5-rc4 commit 8244062ef1 19700 Support dash as /bin/sh in generated Makefile 19756 linux 4.5-rc1 commit 8463833590 19764 stap --tmpdir leaves useless tempdirs behind 19799 deleting from array of aggregate unreliable 19802 bad hash value distribution and horrible performance for large arrays 19812 _stp_target misdirected for pid_namespace containers 19833 monitor mode causes kernel crash when exited immediately 19847 Entering ^C in interactive mode prompt prevents script from running 19873 staprun -o /NO/SUCH/FILE -c CMD imperfect cleanup