2012-08-08 17:14:10

by Arnaldo Carvalho de Melo

Subject: [GIT PULL 00/11] perf/core improvements and fixes

Hi Ingo,

Please consider pulling, on top of my latest pull request,

- Arnaldo

--
1.7.9.2.358.g22243

The following changes since commit 9782243353ec135327a80c76c63464e592949cd1:

perf script: Stop using pevent directly (2012-08-07 23:50:21 -0300)

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux tags/perf-core-for-mingo

for you to fetch changes up to 0076d546b4f9b5c15121c6959d108a83fe43fa9a:

perf scripts python: Add event_analyzing_sample.py as a sample for general event handling (2012-08-08 12:55:38 -0300)

----------------------------------------------------------------
perf/core improvements

. Assorted fixes for Documentation and build in 32 bit, from Robert Richter

. Add support for non-tracepoint events in perf script python, from Feng Tang

Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>

----------------------------------------------------------------
Feng Tang (5):
perf script: Add general python handler to process non-tracepoint events
perf script: Replace "struct thread" with "struct addr_location" as a parameter for "process_event()"
perf scripts python: Pass event/thread/dso name and symbol info to event handler in python
perf scripts python: Add a python library EventClass.py
perf scripts python: Add event_analyzing_sample.py as a sample for general event handling

Robert Richter (6):
perf tools: Fix version file for perf documentation with OUTPUT variable set
perf tools: Fix lib/traceevent build dir with OUTPUT variable set
perf tools: Fix parsing of 64 bit raw config value for 32 bit
tools lib traceevent: Fix cast from pointer to integer for 32 bit
perf list: Update documentation about raw event setup
perf list: Document precise event sampling for AMD IBS

tools/lib/traceevent/event-parse.c | 3 +-
tools/perf/Documentation/Makefile | 6 +-
tools/perf/Documentation/perf-list.txt | 48 +++--
tools/perf/Makefile | 8 +-
tools/perf/builtin-script.c | 5 +-
.../Perf-Trace-Util/lib/Perf/Trace/EventClass.py | 94 ++++++++++
.../perf/scripts/python/event_analyzing_sample.py | 193 ++++++++++++++++++++
tools/perf/util/parse-events.c | 6 +-
tools/perf/util/parse-events.h | 6 +-
tools/perf/util/parse-events.l | 4 +-
tools/perf/util/parse-events.y | 10 +-
.../perf/util/scripting-engines/trace-event-perl.c | 11 +-
.../util/scripting-engines/trace-event-python.c | 85 ++++++++-
tools/perf/util/trace-event-scripting.c | 2 +-
tools/perf/util/trace-event.h | 5 +-
15 files changed, 441 insertions(+), 45 deletions(-)
create mode 100755 tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py
create mode 100644 tools/perf/scripts/python/event_analyzing_sample.py

2012-08-08 17:14:04

by Arnaldo Carvalho de Melo

Subject: [PATCH 02/11] perf tools: Fix lib/traceevent build dir with OUTPUT variable set

From: Robert Richter <[email protected]>

With the OUTPUT variable set the libtraceevent.a file is wrongly built
in the source directory:

+ make -d OUTPUT=/.../.build/perf-user/ DESTDIR=/.../.install/perf-user/
...
Considering target file `../lib/traceevent//libtraceevent.a'.
File `../lib/traceevent//libtraceevent.a' does not exist.
Finished prerequisites of target file `../lib/traceevent//libtraceevent.a'.
Must remake target `../lib/traceevent//libtraceevent.a'.
Invoking recipe from Makefile:837 to update target `../lib/traceevent//libtraceevent.a'.
Putting child 0x703850 (../lib/traceevent//libtraceevent.a) PID 8365 on the chain.
Live child 0x703850 (../lib/traceevent//libtraceevent.a) PID 8365
SUBDIR ../lib/traceevent/

$ git clean -nxd
Would remove tools/lib/traceevent/.event-parse.d
Would remove tools/lib/traceevent/.parse-filter.d
Would remove tools/lib/traceevent/.parse-utils.d
Would remove tools/lib/traceevent/.trace-seq.d
Would remove tools/lib/traceevent/event-parse.o
Would remove tools/lib/traceevent/libtraceevent.a
Would remove tools/lib/traceevent/parse-filter.o
Would remove tools/lib/traceevent/parse-utils.o
Would remove tools/lib/traceevent/trace-seq.o

This patch fixes this.

Note: Though this should already work with O=$outputdir we better use
the OUTPUT variable directly.

Signed-off-by: Robert Richter <[email protected]>
Cc: Ingo Molnar <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
tools/perf/Makefile | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/tools/perf/Makefile b/tools/perf/Makefile
index 35655c3..2d4bf6e 100644
--- a/tools/perf/Makefile
+++ b/tools/perf/Makefile
@@ -186,10 +186,10 @@ SCRIPTS = $(patsubst %.sh,%,$(SCRIPT_SH))

TRACE_EVENT_DIR = ../lib/traceevent/

-ifeq ("$(origin O)", "command line")
- TE_PATH=$(OUTPUT)/
+ifneq ($(OUTPUT),)
+ TE_PATH=$(OUTPUT)
else
- TE_PATH=$(TRACE_EVENT_DIR)/
+ TE_PATH=$(TRACE_EVENT_DIR)
endif

LIBTRACEEVENT = $(TE_PATH)libtraceevent.a
@@ -842,7 +842,7 @@ $(LIB_FILE): $(LIB_OBJS)

# libtraceevent.a
$(LIBTRACEEVENT):
- $(QUIET_SUBDIR0)$(TRACE_EVENT_DIR) $(QUIET_SUBDIR1) $(COMMAND_O) libtraceevent.a
+ $(QUIET_SUBDIR0)$(TRACE_EVENT_DIR) $(QUIET_SUBDIR1) O=$(OUTPUT) libtraceevent.a

help:
@echo 'Perf make targets:'
--
1.7.9.2.358.g22243

2012-08-08 17:14:15

by Arnaldo Carvalho de Melo

Subject: [PATCH 10/11] perf scripts python: Add a python library EventClass.py

From: Feng Tang <[email protected]>

This library defines several class types for perf events which could
help to better analyze the event samples. Currently there are just a few
classes, PerfEvent is the base class for all perf events, PebsEvent is
a HW base Intel x86 PEBS event, and user could add more SW/HW event
classes based on requriements.

Signed-off-by: Feng Tang <[email protected]>
Cc: Andi Kleen <[email protected]>
Cc: David Ahern <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Robert Richter <[email protected]>
Cc: Stephane Eranian <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
.../Perf-Trace-Util/lib/Perf/Trace/EventClass.py | 94 ++++++++++++++++++++
1 file changed, 94 insertions(+)
create mode 100755 tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py

diff --git a/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py b/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py
new file mode 100755
index 0000000..6372431
--- /dev/null
+++ b/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py
@@ -0,0 +1,94 @@
+# EventClass.py
+#
+# This is a libray defining some events typs classes, which could
+# be used by other scripts to analyzing the perf samples.
+#
+# Currently there are just a few classes defined for examples,
+# PerfEvent is the base class for all perf event sample, PebsEvent
+# is a HW base Intel x86 PEBS event, and user could add more SW/HW
+# event classes based on requriements.
+
+import struct
+
+# Event types, user could add more here
+EVTYPE_GENERIC = 0
+EVTYPE_PEBS = 1 # Basic PEBS event
+EVTYPE_PEBS_LL = 2 # PEBS event with load latency info
+EVTYPE_IBS = 3
+
+#
+# Currently we don't have good way to tell the event type, but by
+# the size of raw buffer, raw PEBS event with load latency data's
+# size is 176 bytes, while the pure PEBS event's size is 144 bytes.
+#
+def create_event(name, comm, dso, symbol, raw_buf):
+ if (len(raw_buf) == 144):
+ event = PebsEvent(name, comm, dso, symbol, raw_buf)
+ elif (len(raw_buf) == 176):
+ event = PebsNHM(name, comm, dso, symbol, raw_buf)
+ else:
+ event = PerfEvent(name, comm, dso, symbol, raw_buf)
+
+ return event
+
+class PerfEvent(object):
+ event_num = 0
+ def __init__(self, name, comm, dso, symbol, raw_buf, ev_type=EVTYPE_GENERIC):
+ self.name = name
+ self.comm = comm
+ self.dso = dso
+ self.symbol = symbol
+ self.raw_buf = raw_buf
+ self.ev_type = ev_type
+ PerfEvent.event_num += 1
+
+ def show(self):
+ print "PMU event: name=%12s, symbol=%24s, comm=%8s, dso=%12s" % (self.name, self.symbol, self.comm, self.dso)
+
+#
+# Basic Intel PEBS (Precise Event-based Sampling) event, whose raw buffer
+# contains the context info when that event happened: the EFLAGS and
+# linear IP info, as well as all the registers.
+#
+class PebsEvent(PerfEvent):
+ pebs_num = 0
+ def __init__(self, name, comm, dso, symbol, raw_buf, ev_type=EVTYPE_PEBS):
+ tmp_buf=raw_buf[0:80]
+ flags, ip, ax, bx, cx, dx, si, di, bp, sp = struct.unpack('QQQQQQQQQQ', tmp_buf)
+ self.flags = flags
+ self.ip = ip
+ self.ax = ax
+ self.bx = bx
+ self.cx = cx
+ self.dx = dx
+ self.si = si
+ self.di = di
+ self.bp = bp
+ self.sp = sp
+
+ PerfEvent.__init__(self, name, comm, dso, symbol, raw_buf, ev_type)
+ PebsEvent.pebs_num += 1
+ del tmp_buf
+
+#
+# Intel Nehalem and Westmere support PEBS plus Load Latency info which lie
+# in the four 64 bit words write after the PEBS data:
+# Status: records the IA32_PERF_GLOBAL_STATUS register value
+# DLA: Data Linear Address (EIP)
+# DSE: Data Source Encoding, where the latency happens, hit or miss
+# in L1/L2/L3 or IO operations
+# LAT: the actual latency in cycles
+#
+class PebsNHM(PebsEvent):
+ pebs_nhm_num = 0
+ def __init__(self, name, comm, dso, symbol, raw_buf, ev_type=EVTYPE_PEBS_LL):
+ tmp_buf=raw_buf[144:176]
+ status, dla, dse, lat = struct.unpack('QQQQ', tmp_buf)
+ self.status = status
+ self.dla = dla
+ self.dse = dse
+ self.lat = lat
+
+ PebsEvent.__init__(self, name, comm, dso, symbol, raw_buf, ev_type)
+ PebsNHM.pebs_nhm_num += 1
+ del tmp_buf
--
1.7.9.2.358.g22243

2012-08-08 17:14:27

by Arnaldo Carvalho de Melo

Subject: [PATCH 08/11] perf script: Replace "struct thread" with "struct addr_location" as a parameter for "process_event()"

From: Feng Tang <[email protected]>

Both perl and python script start processing events other than trace
points, and it's useful to pass the resolved symbol and the dso info to
the event handler in script for better analysis and statistics.

Struct thread is already a member of struct addr_location, using
addr_location will keep the thread info, while providing additional
symbol and dso info if exist, so that the script itself doesn't need to
bother to do the symbol resolving and dso searching work.

Tested-by: David Ahern <[email protected]>
Signed-off-by: Feng Tang <[email protected]>
Acked-by: David Ahern <[email protected]>
Cc: Andi Kleen <[email protected]>
Cc: David Ahern <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Robert Richter <[email protected]>
Cc: Stephane Eranian <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
tools/perf/builtin-script.c | 5 +++--
.../perf/util/scripting-engines/trace-event-perl.c | 11 ++++++-----
.../util/scripting-engines/trace-event-python.c | 13 +++++++------
tools/perf/util/trace-event-scripting.c | 2 +-
tools/perf/util/trace-event.h | 5 +++--
5 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 6425612..30a9cb8 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -396,9 +396,10 @@ static void print_sample_bts(union perf_event *event,

static void process_event(union perf_event *event, struct perf_sample *sample,
struct perf_evsel *evsel, struct machine *machine,
- struct thread *thread)
+ struct addr_location *al)
{
struct perf_event_attr *attr = &evsel->attr;
+ struct thread *thread = al->thread;

if (output[attr->type].fields == 0)
return;
@@ -511,7 +512,7 @@ static int process_sample_event(struct perf_tool *tool __used,
if (cpu_list && !test_bit(sample->cpu, cpu_bitmap))
return 0;

- scripting_ops->process_event(event, sample, evsel, machine, thread);
+ scripting_ops->process_event(event, sample, evsel, machine, &al);

evsel->hists.stats.total_period += sample->period;
return 0;
diff --git a/tools/perf/util/scripting-engines/trace-event-perl.c b/tools/perf/util/scripting-engines/trace-event-perl.c
index 52580d0..d280010 100644
--- a/tools/perf/util/scripting-engines/trace-event-perl.c
+++ b/tools/perf/util/scripting-engines/trace-event-perl.c
@@ -261,7 +261,7 @@ static void perl_process_tracepoint(union perf_event *perf_event __unused,
struct perf_sample *sample,
struct perf_evsel *evsel,
struct machine *machine __unused,
- struct thread *thread)
+ struct addr_location *al)
{
struct format_field *field;
static char handler[256];
@@ -272,6 +272,7 @@ static void perl_process_tracepoint(union perf_event *perf_event __unused,
int cpu = sample->cpu;
void *data = sample->raw_data;
unsigned long long nsecs = sample->time;
+ struct thread *thread = al->thread;
char *comm = thread->comm;

dSP;
@@ -349,7 +350,7 @@ static void perl_process_event_generic(union perf_event *event,
struct perf_sample *sample,
struct perf_evsel *evsel,
struct machine *machine __unused,
- struct thread *thread __unused)
+ struct addr_location *al __unused)
{
dSP;

@@ -375,10 +376,10 @@ static void perl_process_event(union perf_event *event,
struct perf_sample *sample,
struct perf_evsel *evsel,
struct machine *machine,
- struct thread *thread)
+ struct addr_location *al)
{
- perl_process_tracepoint(event, sample, evsel, machine, thread);
- perl_process_event_generic(event, sample, evsel, machine, thread);
+ perl_process_tracepoint(event, sample, evsel, machine, al);
+ perl_process_event_generic(event, sample, evsel, machine, al);
}

static void run_start_sub(void)
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index b9010d8..24711b3 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -225,7 +225,7 @@ static void python_process_tracepoint(union perf_event *perf_event __unused,
struct perf_sample *sample,
struct perf_evsel *evsel,
struct machine *machine __unused,
- struct thread *thread)
+ struct addr_location *al)
{
PyObject *handler, *retval, *context, *t, *obj, *dict = NULL;
static char handler_name[256];
@@ -238,6 +238,7 @@ static void python_process_tracepoint(union perf_event *perf_event __unused,
int cpu = sample->cpu;
void *data = sample->raw_data;
unsigned long long nsecs = sample->time;
+ struct thread *thread = al->thread;
char *comm = thread->comm;

t = PyTuple_New(MAX_FIELDS);
@@ -342,7 +343,7 @@ static void python_process_general_event(union perf_event *perf_event __unused,
struct perf_sample *sample,
struct perf_evsel *evsel,
struct machine *machine __unused,
- struct thread *thread __unused)
+ struct addr_location *al __unused)
{
PyObject *handler, *retval, *t;
static char handler_name[64];
@@ -361,7 +362,7 @@ static void python_process_general_event(union perf_event *perf_event __unused,
goto exit;
}

- /* Pass 3 parameters: event_attr, perf_sample, raw data */
+ /* Pass 4 parameters: event_attr, perf_sample, raw data, thread name */
PyTuple_SetItem(t, n++, PyString_FromStringAndSize((void *)&evsel->attr, sizeof(evsel->attr)));
PyTuple_SetItem(t, n++, PyString_FromStringAndSize((void *)sample, sizeof(*sample)));
PyTuple_SetItem(t, n++, PyString_FromStringAndSize(data, sample->raw_size));
@@ -380,17 +381,17 @@ static void python_process_event(union perf_event *perf_event,
struct perf_sample *sample,
struct perf_evsel *evsel,
struct machine *machine,
- struct thread *thread)
+ struct addr_location *al)
{
switch (evsel->attr.type) {
case PERF_TYPE_TRACEPOINT:
python_process_tracepoint(perf_event, sample, evsel,
- machine, thread);
+ machine, al);
break;
/* Reserve for future process_hw/sw/raw APIs */
default:
python_process_general_event(perf_event, sample, evsel,
- machine, thread);
+ machine, al);
}
}

diff --git a/tools/perf/util/trace-event-scripting.c b/tools/perf/util/trace-event-scripting.c
index aceb8ee..302ff26 100644
--- a/tools/perf/util/trace-event-scripting.c
+++ b/tools/perf/util/trace-event-scripting.c
@@ -39,7 +39,7 @@ static void process_event_unsupported(union perf_event *event __unused,
struct perf_sample *sample __unused,
struct perf_evsel *evsel __unused,
struct machine *machine __unused,
- struct thread *thread __unused)
+ struct addr_location *al __unused)
{
}

diff --git a/tools/perf/util/trace-event.h b/tools/perf/util/trace-event.h
index cee1635..7575dfd 100644
--- a/tools/perf/util/trace-event.h
+++ b/tools/perf/util/trace-event.h
@@ -9,7 +9,6 @@ struct machine;
struct perf_sample;
union perf_event;
struct perf_tool;
-struct thread;

extern int header_page_size_size;
extern int header_page_ts_size;
@@ -76,6 +75,8 @@ struct tracing_data *tracing_data_get(struct list_head *pattrs,
void tracing_data_put(struct tracing_data *tdata);


+struct addr_location;
+
struct scripting_ops {
const char *name;
int (*start_script) (const char *script, int argc, const char **argv);
@@ -84,7 +85,7 @@ struct scripting_ops {
struct perf_sample *sample,
struct perf_evsel *evsel,
struct machine *machine,
- struct thread *thread);
+ struct addr_location *al);
int (*generate_script) (struct pevent *pevent, const char *outfile);
};

--
1.7.9.2.358.g22243

2012-08-08 17:14:24

by Arnaldo Carvalho de Melo

Subject: [PATCH 11/11] perf scripts python: Add event_analyzing_sample.py as a sample for general event handling

From: Feng Tang <[email protected]>

Currently only trace point events are supported in perf/python script,
the first 3 patches of this serie add the support for all types of
events. This script is just a simple sample to show how to gather the
basic information of the events and analyze them.

This script will create one object for each event sample and insert them
into a table in a database, then leverage the simple SQL commands to
sort/group them. User can modify or write their brand new functions
according to their specific requirment.

Here is the sample of how to use the script:

$ perf record -a tree
$ perf script -s process_event.py

There is 100 records in gen_events table
Statistics about the general events grouped by thread/symbol/dso:

comm number histgram
==========================================
swapper 56 ######
tree 20 #####
perf 10 ####
sshd 8 ####
kworker/7:2 4 ###
ksoftirqd/7 1 #
plugin-containe 1 #

symbol number histgram
==========================================================
native_write_msr_safe 40 ######
__lock_acquire 8 ####
ftrace_graph_caller 4 ###
prepare_ftrace_return 4 ###
intel_idle 3 ##
native_sched_clock 3 ##
Unknown_symbol 2 ##
do_softirq 2 ##
lock_release 2 ##
lock_release_holdtime 2 ##
trace_graph_entry 2 ##
_IO_putc 1 #
__d_lookup_rcu 1 #
__do_fault 1 #
__schedule 1 #
_raw_spin_lock 1 #
delay_tsc 1 #
generic_exec_single 1 #
generic_fillattr 1 #

dso number histgram
==================================================================
[kernel.kallsyms] 95 #######
/lib/libc-2.12.1.so 5 ###

Signed-off-by: Feng Tang <[email protected]>
Cc: Andi Kleen <[email protected]>
Cc: David Ahern <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Robert Richter <[email protected]>
Cc: Stephane Eranian <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
.../perf/scripts/python/event_analyzing_sample.py | 193 ++++++++++++++++++++
1 file changed, 193 insertions(+)
create mode 100644 tools/perf/scripts/python/event_analyzing_sample.py

diff --git a/tools/perf/scripts/python/event_analyzing_sample.py b/tools/perf/scripts/python/event_analyzing_sample.py
new file mode 100644
index 0000000..46f05aa
--- /dev/null
+++ b/tools/perf/scripts/python/event_analyzing_sample.py
@@ -0,0 +1,193 @@
+# process_event.py: general event handler in python
+#
+# Current perf report is alreay very powerful with the anotation integrated,
+# and this script is not trying to be as powerful as perf report, but
+# providing end user/developer a flexible way to analyze the events other
+# than trace points.
+#
+# The 2 database related functions in this script just show how to gather
+# the basic information, and users can modify and write their own functions
+# according to their specific requirment.
+#
+# The first sample "show_general_events" just does a baisc grouping for all
+# generic events with the help of sqlite, and the 2nd one "show_pebs_ll" is
+# for a x86 HW PMU event: PEBS with load latency data.
+#
+
+import os
+import sys
+import math
+import struct
+import sqlite3
+
+sys.path.append(os.environ['PERF_EXEC_PATH'] + \
+ '/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
+
+from perf_trace_context import *
+from EventClass import *
+
+#
+# If the perf.data has a big number of samples, then the insert operation
+# will be very time consuming (about 10+ minutes for 10000 samples) if the
+# .db database is on disk. Move the .db file to RAM based FS to speedup
+# the handling, which will cut the time down to several seconds.
+#
+con = sqlite3.connect("/dev/shm/perf.db")
+con.isolation_level = None
+
+def trace_begin():
+ print "In trace_begin:\n"
+
+ #
+ # Will create several tables at the start, pebs_ll is for PEBS data with
+ # load latency info, while gen_events is for general event.
+ #
+ con.execute("""
+ create table if not exists gen_events (
+ name text,
+ symbol text,
+ comm text,
+ dso text
+ );""")
+ con.execute("""
+ create table if not exists pebs_ll (
+ name text,
+ symbol text,
+ comm text,
+ dso text,
+ flags integer,
+ ip integer,
+ status integer,
+ dse integer,
+ dla integer,
+ lat integer
+ );""")
+
+#
+# Create and insert event object to a database so that user could
+# do more analysis with simple database commands.
+#
+def process_event(param_dict):
+ event_attr = param_dict["attr"]
+ sample = param_dict["sample"]
+ raw_buf = param_dict["raw_buf"]
+ comm = param_dict["comm"]
+ name = param_dict["ev_name"]
+
+ # Symbol and dso info are not always resolved
+ if (param_dict.has_key("dso")):
+ dso = param_dict["dso"]
+ else:
+ dso = "Unknown_dso"
+
+ if (param_dict.has_key("symbol")):
+ symbol = param_dict["symbol"]
+ else:
+ symbol = "Unknown_symbol"
+
+ # Creat the event object and insert it to the right table in database
+ event = create_event(name, comm, dso, symbol, raw_buf)
+ insert_db(event)
+
+def insert_db(event):
+ if event.ev_type == EVTYPE_GENERIC:
+ con.execute("insert into gen_events values(?, ?, ?, ?)",
+ (event.name, event.symbol, event.comm, event.dso))
+ elif event.ev_type == EVTYPE_PEBS_LL:
+ event.ip &= 0x7fffffffffffffff
+ event.dla &= 0x7fffffffffffffff
+ con.execute("insert into pebs_ll values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
+ (event.name, event.symbol, event.comm, event.dso, event.flags,
+ event.ip, event.status, event.dse, event.dla, event.lat))
+
+def trace_end():
+ print "In trace_end:\n"
+ # We show the basic info for the 2 type of event classes
+ show_general_events()
+ show_pebs_ll()
+ con.close()
+
+#
+# As the event number may be very big, so we can't use linear way
+# to show the histgram in real number, but use a log2 algorithm.
+#
+
+def num2sym(num):
+ # Each number will have at least one '#'
+ snum = '#' * (int)(math.log(num, 2) + 1)
+ return snum
+
+def show_general_events():
+
+ # Check the total record number in the table
+ count = con.execute("select count(*) from gen_events")
+ for t in count:
+ print "There is %d records in gen_events table" % t[0]
+ if t[0] == 0:
+ return
+
+ print "Statistics about the general events grouped by thread/symbol/dso: \n"
+
+ # Group by thread
+ commq = con.execute("select comm, count(comm) from gen_events group by comm order by -count(comm)")
+ print "\n%16s %8s %16s\n%s" % ("comm", "number", "histgram", "="*42)
+ for row in commq:
+ print "%16s %8d %s" % (row[0], row[1], num2sym(row[1]))
+
+ # Group by symbol
+ print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histgram", "="*58)
+ symbolq = con.execute("select symbol, count(symbol) from gen_events group by symbol order by -count(symbol)")
+ for row in symbolq:
+ print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))
+
+ # Group by dso
+ print "\n%40s %8s %16s\n%s" % ("dso", "number", "histgram", "="*74)
+ dsoq = con.execute("select dso, count(dso) from gen_events group by dso order by -count(dso)")
+ for row in dsoq:
+ print "%40s %8d %s" % (row[0], row[1], num2sym(row[1]))
+
+#
+# This function just shows the basic info, and we could do more with the
+# data in the tables, like checking the function parameters when some
+# big latency events happen.
+#
+def show_pebs_ll():
+
+ count = con.execute("select count(*) from pebs_ll")
+ for t in count:
+ print "There is %d records in pebs_ll table" % t[0]
+ if t[0] == 0:
+ return
+
+ print "Statistics about the PEBS Load Latency events grouped by thread/symbol/dse/latency: \n"
+
+ # Group by thread
+ commq = con.execute("select comm, count(comm) from pebs_ll group by comm order by -count(comm)")
+ print "\n%16s %8s %16s\n%s" % ("comm", "number", "histgram", "="*42)
+ for row in commq:
+ print "%16s %8d %s" % (row[0], row[1], num2sym(row[1]))
+
+ # Group by symbol
+ print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histgram", "="*58)
+ symbolq = con.execute("select symbol, count(symbol) from pebs_ll group by symbol order by -count(symbol)")
+ for row in symbolq:
+ print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))
+
+ # Group by dse
+ dseq = con.execute("select dse, count(dse) from pebs_ll group by dse order by -count(dse)")
+ print "\n%32s %8s %16s\n%s" % ("dse", "number", "histgram", "="*58)
+ for row in dseq:
+ print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))
+
+ # Group by latency
+ latq = con.execute("select lat, count(lat) from pebs_ll group by lat order by lat")
+ print "\n%32s %8s %16s\n%s" % ("latency", "number", "histgram", "="*58)
+ for row in latq:
+ print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))
+
+def trace_unhandled(event_name, context, event_fields_dict):
+ print ' '.join(['%s=%s'%(k,str(v))for k,v in sorted(event_fields_dict.items())])
+
+def print_header(event_name, cpu, secs, nsecs, pid, comm):
+ print "%-20s %5u %05u.%09u %8u %-20s " % \
+ (event_name, cpu, secs, nsecs, pid, comm),
--
1.7.9.2.358.g22243

2012-08-08 17:15:14

by Arnaldo Carvalho de Melo

Subject: [PATCH 09/11] perf scripts python: Pass event/thread/dso name and symbol info to event handler in python

From: Feng Tang <[email protected]>

Also as suggested by Arnaldo, pack all these parameters to a dictionary,
which is more expandable for adding new parameters while keeping the
compatibility for old scripts.

Signed-off-by: Feng Tang <[email protected]>
Cc: Andi Kleen <[email protected]>
Cc: David Ahern <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Robert Richter <[email protected]>
Cc: Stephane Eranian <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
.../util/scripting-engines/trace-event-python.c | 35 ++++++++++++++++----
1 file changed, 29 insertions(+), 6 deletions(-)

diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 24711b3..7e3f576 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -345,15 +345,23 @@ static void python_process_general_event(union perf_event *perf_event __unused,
struct machine *machine __unused,
struct addr_location *al __unused)
{
- PyObject *handler, *retval, *t;
+ PyObject *handler, *retval, *t, *dict;
static char handler_name[64];
unsigned n = 0;
- void *data = sample->raw_data;
+ struct thread *thread = al->thread;

+ /*
+ * Use the MAX_FIELDS to make the function expandable, though
+ * currently there is only one itme for the tuple.
+ */
t = PyTuple_New(MAX_FIELDS);
if (!t)
Py_FatalError("couldn't create Python tuple");

+ dict = PyDict_New();
+ if (!dict)
+ Py_FatalError("couldn't create Python dictionary");
+
snprintf(handler_name, sizeof(handler_name), "%s", "process_event");

handler = PyDict_GetItemString(main_dict, handler_name);
@@ -362,11 +370,25 @@ static void python_process_general_event(union perf_event *perf_event __unused,
goto exit;
}

- /* Pass 4 parameters: event_attr, perf_sample, raw data, thread name */
- PyTuple_SetItem(t, n++, PyString_FromStringAndSize((void *)&evsel->attr, sizeof(evsel->attr)));
- PyTuple_SetItem(t, n++, PyString_FromStringAndSize((void *)sample, sizeof(*sample)));
- PyTuple_SetItem(t, n++, PyString_FromStringAndSize(data, sample->raw_size));
+ PyDict_SetItemString(dict, "ev_name", PyString_FromString(perf_evsel__name(evsel)));
+ PyDict_SetItemString(dict, "attr", PyString_FromStringAndSize(
+ (const char *)&evsel->attr, sizeof(evsel->attr)));
+ PyDict_SetItemString(dict, "sample", PyString_FromStringAndSize(
+ (const char *)sample, sizeof(*sample)));
+ PyDict_SetItemString(dict, "raw_buf", PyString_FromStringAndSize(
+ (const char *)sample->raw_data, sample->raw_size));
+ PyDict_SetItemString(dict, "comm",
+ PyString_FromString(thread->comm));
+ if (al->map) {
+ PyDict_SetItemString(dict, "dso",
+ PyString_FromString(al->map->dso->name));
+ }
+ if (al->sym) {
+ PyDict_SetItemString(dict, "symbol",
+ PyString_FromString(al->sym->name));
+ }

+ PyTuple_SetItem(t, n++, dict);
if (_PyTuple_Resize(&t, n) == -1)
Py_FatalError("error resizing Python tuple");

@@ -374,6 +396,7 @@ static void python_process_general_event(union perf_event *perf_event __unused,
if (retval == NULL)
handler_call_die(handler_name);
exit:
+ Py_DECREF(dict);
Py_DECREF(t);
}

--
1.7.9.2.358.g22243

2012-08-08 17:15:53

by Arnaldo Carvalho de Melo

Subject: [PATCH 07/11] perf script: Add general python handler to process non-tracepoint events

From: Feng Tang <[email protected]>

This patch just follows Robert Richter's idea and the commit 37a058ea0
"perf script: Add generic perl handler to process events"
to similarly add a python handler for general events other than tracepoints.

For non-tracepoint events, this patch will try to find a function named
"process_event" in the python script, and pass the event attribute,
perf_sample, raw_data in format of raw string. And the python script can
use "struct" module's unpack function to disasemble the needed info and process.

Signed-off-by: Feng Tang <[email protected]>
Cc: Andi Kleen <[email protected]>
Cc: David Ahern <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Robert Richter <[email protected]>
Cc: Stephane Eranian <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
[ committer note: Fixed up wrt da37896, i.e. pevent parm in script event handlers ]
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
.../util/scripting-engines/trace-event-python.c | 59 +++++++++++++++++++-
1 file changed, 58 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index df7d33d..b9010d8 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -32,6 +32,7 @@
#include "../event.h"
#include "../thread.h"
#include "../trace-event.h"
+#include "../evsel.h"

PyMODINIT_FUNC initperf_trace_context(void);

@@ -220,7 +221,7 @@ static inline struct event_format *find_cache_event(struct perf_evsel *evsel)
return event;
}

-static void python_process_event(union perf_event *perf_event __unused,
+static void python_process_tracepoint(union perf_event *perf_event __unused,
struct perf_sample *sample,
struct perf_evsel *evsel,
struct machine *machine __unused,
@@ -337,6 +338,62 @@ static void python_process_event(union perf_event *perf_event __unused,
Py_DECREF(t);
}

+static void python_process_general_event(union perf_event *perf_event __unused,
+ struct perf_sample *sample,
+ struct perf_evsel *evsel,
+ struct machine *machine __unused,
+ struct thread *thread __unused)
+{
+ PyObject *handler, *retval, *t;
+ static char handler_name[64];
+ unsigned n = 0;
+ void *data = sample->raw_data;
+
+ t = PyTuple_New(MAX_FIELDS);
+ if (!t)
+ Py_FatalError("couldn't create Python tuple");
+
+ snprintf(handler_name, sizeof(handler_name), "%s", "process_event");
+
+ handler = PyDict_GetItemString(main_dict, handler_name);
+ if (handler && !PyCallable_Check(handler)) {
+ handler = NULL;
+ goto exit;
+ }
+
+ /* Pass 3 parameters: event_attr, perf_sample, raw data */
+ PyTuple_SetItem(t, n++, PyString_FromStringAndSize((void *)&evsel->attr, sizeof(evsel->attr)));
+ PyTuple_SetItem(t, n++, PyString_FromStringAndSize((void *)sample, sizeof(*sample)));
+ PyTuple_SetItem(t, n++, PyString_FromStringAndSize(data, sample->raw_size));
+
+ if (_PyTuple_Resize(&t, n) == -1)
+ Py_FatalError("error resizing Python tuple");
+
+ retval = PyObject_CallObject(handler, t);
+ if (retval == NULL)
+ handler_call_die(handler_name);
+exit:
+ Py_DECREF(t);
+}
+
+static void python_process_event(union perf_event *perf_event,
+ struct perf_sample *sample,
+ struct perf_evsel *evsel,
+ struct machine *machine,
+ struct thread *thread)
+{
+ switch (evsel->attr.type) {
+ case PERF_TYPE_TRACEPOINT:
+ python_process_tracepoint(perf_event, sample, evsel,
+ machine, thread);
+ break;
+ /* Reserve for future process_hw/sw/raw APIs */
+ default:
+ python_process_general_event(perf_event, sample, evsel,
+ machine, thread);
+ }
+}
+
static int run_start_sub(void)
{
PyObject *handler, *retval;
--
1.7.9.2.358.g22243

2012-08-08 17:14:01

by Arnaldo Carvalho de Melo

Subject: [PATCH 04/11] tools lib traceevent: Fix cast from pointer to integer for 32 bit

From: Robert Richter <[email protected]>

Fixing the integer cast reported by the following warning:

tools/lib/traceevent/event-parse.c:3488:14: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]

Signed-off-by: Robert Richter <[email protected]>
Cc: Ingo Molnar <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
tools/lib/traceevent/event-parse.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/lib/traceevent/event-parse.c b/tools/lib/traceevent/event-parse.c
index 5f34aa3..b7c2c49 100644
--- a/tools/lib/traceevent/event-parse.c
+++ b/tools/lib/traceevent/event-parse.c
@@ -31,6 +31,7 @@
#include <stdarg.h>
#include <ctype.h>
#include <errno.h>
+#include <stdint.h>

#include "event-parse.h"
#include "event-utils.h"
@@ -3485,7 +3486,7 @@ process_defined_func(struct trace_seq *s, void *data, int size,
if (!string->str)
die("malloc str");

- args[i] = (unsigned long long)string->str;
+ args[i] = (uintptr_t)string->str;
strings = string;
trace_seq_destroy(&str);
break;
--
1.7.9.2.358.g22243

2012-08-08 17:16:22

by Arnaldo Carvalho de Melo

Subject: [PATCH 03/11] perf tools: Fix parsing of 64 bit raw config value for 32 bit

From: Robert Richter <[email protected]>

perf record fails on 32 bit with:

invalid or unsupported event: 'r40000F7E0'

Fixing this by parsing 64 bit num values.

Signed-off-by: Robert Richter <[email protected]>
Cc: Ingo Molnar <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
tools/perf/util/parse-events.c | 6 +++---
tools/perf/util/parse-events.h | 6 +++---
tools/perf/util/parse-events.l | 4 ++--
tools/perf/util/parse-events.y | 10 +++++-----
4 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index 74a5af4..8bdfa3e 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -551,7 +551,7 @@ static int config_attr(struct perf_event_attr *attr,
}

int parse_events_add_numeric(struct list_head **list, int *idx,
- unsigned long type, unsigned long config,
+ u32 type, u64 config,
struct list_head *head_config)
{
struct perf_event_attr attr;
@@ -1005,7 +1005,7 @@ int parse_events__is_hardcoded_term(struct parse_events__term *term)

static int new_term(struct parse_events__term **_term, int type_val,
int type_term, char *config,
- char *str, long num)
+ char *str, u64 num)
{
struct parse_events__term *term;

@@ -1034,7 +1034,7 @@ static int new_term(struct parse_events__term **_term, int type_val,
}

int parse_events__term_num(struct parse_events__term **term,
- int type_term, char *config, long num)
+ int type_term, char *config, u64 num)
{
return new_term(term, PARSE_EVENTS__TERM_TYPE_NUM, type_term,
config, NULL, num);
diff --git a/tools/perf/util/parse-events.h b/tools/perf/util/parse-events.h
index ee9c218..163aad4 100644
--- a/tools/perf/util/parse-events.h
+++ b/tools/perf/util/parse-events.h
@@ -55,7 +55,7 @@ struct parse_events__term {
char *config;
union {
char *str;
- long num;
+ u64 num;
} val;
int type_val;
int type_term;
@@ -73,7 +73,7 @@ struct parse_events_data__terms {

int parse_events__is_hardcoded_term(struct parse_events__term *term);
int parse_events__term_num(struct parse_events__term **_term,
- int type_term, char *config, long num);
+ int type_term, char *config, u64 num);
int parse_events__term_str(struct parse_events__term **_term,
int type_term, char *config, char *str);
int parse_events__term_clone(struct parse_events__term **new,
@@ -83,7 +83,7 @@ int parse_events_modifier(struct list_head *list, char *str);
int parse_events_add_tracepoint(struct list_head **list, int *idx,
char *sys, char *event);
int parse_events_add_numeric(struct list_head **list, int *idx,
- unsigned long type, unsigned long config,
+ u32 type, u64 config,
struct list_head *head_config);
int parse_events_add_cache(struct list_head **list, int *idx,
char *type, char *op_result1, char *op_result2);
diff --git a/tools/perf/util/parse-events.l b/tools/perf/util/parse-events.l
index 384ca74..e4abdf2 100644
--- a/tools/perf/util/parse-events.l
+++ b/tools/perf/util/parse-events.l
@@ -15,10 +15,10 @@ YYSTYPE *parse_events_get_lval(yyscan_t yyscanner);

static int __value(YYSTYPE *yylval, char *str, int base, int token)
{
- long num;
+ u64 num;

errno = 0;
- num = strtoul(str, NULL, base);
+ num = strtoull(str, NULL, base);
if (errno)
return PE_ERROR;

diff --git a/tools/perf/util/parse-events.y b/tools/perf/util/parse-events.y
index 2bc5fbf..423d331 100644
--- a/tools/perf/util/parse-events.y
+++ b/tools/perf/util/parse-events.y
@@ -57,7 +57,7 @@ do { \
%union
{
char *str;
- unsigned long num;
+ u64 num;
struct list_head *head;
struct parse_events__term *term;
}
@@ -207,7 +207,7 @@ PE_VALUE ':' PE_VALUE
struct parse_events_data__events *data = _data;
struct list_head *list = NULL;

- ABORT_ON(parse_events_add_numeric(&list, &data->idx, $1, $3, NULL));
+ ABORT_ON(parse_events_add_numeric(&list, &data->idx, (u32)$1, $3, NULL));
$$ = list;
}

@@ -282,7 +282,7 @@ PE_TERM '=' PE_NAME
{
struct parse_events__term *term;

- ABORT_ON(parse_events__term_str(&term, $1, NULL, $3));
+ ABORT_ON(parse_events__term_str(&term, (int)$1, NULL, $3));
$$ = term;
}
|
@@ -290,7 +290,7 @@ PE_TERM '=' PE_VALUE
{
struct parse_events__term *term;

- ABORT_ON(parse_events__term_num(&term, $1, NULL, $3));
+ ABORT_ON(parse_events__term_num(&term, (int)$1, NULL, $3));
$$ = term;
}
|
@@ -298,7 +298,7 @@ PE_TERM
{
struct parse_events__term *term;

- ABORT_ON(parse_events__term_num(&term, $1, NULL, 1));
+ ABORT_ON(parse_events__term_num(&term, (int)$1, NULL, 1));
$$ = term;
}

--
1.7.9.2.358.g22243

2012-08-08 17:16:42

by Arnaldo Carvalho de Melo

Subject: [PATCH 05/11] perf list: Update documentation about raw event setup

From: Robert Richter <[email protected]>

It was missing that only certain bit fields are passed to the config
value which confused users. Updating it.

Signed-off-by: Robert Richter <[email protected]>
Cc: Ingo Molnar <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
tools/perf/Documentation/perf-list.txt | 6 ++++++
1 file changed, 6 insertions(+)

diff --git a/tools/perf/Documentation/perf-list.txt b/tools/perf/Documentation/perf-list.txt
index ddc2252..232be51 100644
--- a/tools/perf/Documentation/perf-list.txt
+++ b/tools/perf/Documentation/perf-list.txt
@@ -15,6 +15,7 @@ DESCRIPTION
This command displays the symbolic event types which can be selected in the
various perf commands with the -e option.

+[[EVENT_MODIFIERS]]
EVENT MODIFIERS
---------------

@@ -44,6 +45,11 @@ layout of IA32_PERFEVTSELx MSRs (see [Intel® 64 and IA-32 Architectures Softwar
of IA32_PERFEVTSELx MSRs) or AMD's PerfEvtSeln (see [AMD64 Architecture Programmer’s Manual Volume 2: System Programming], Page 344,
Figure 13-7 Performance Event-Select Register (PerfEvtSeln)).

+Note: Only the following bit fields can be set in x86 counter
+registers: event, umask, edge, inv, cmask. Esp. guest/host only and
+OS/user mode flags must be setup using <<EVENT_MODIFIERS, EVENT
+MODIFIERS>>.
+
Example:

If the Intel docs for a QM720 Core i7 describe an event as:
--
1.7.9.2.358.g22243

2012-08-08 17:13:59

by Arnaldo Carvalho de Melo

Subject: [PATCH 01/11] perf tools: Fix version file for perf documentation with OUTPUT variable set

From: Robert Richter <[email protected]>

Fixes the following:

+ make OUTPUT=/.../.build/perf-user/ DESTDIR=/.../.install/perf-user/ man install-man
make -C Documentation man
make[1]: Entering directory `/.../.source/linux.perf/tools/perf/Documentation'
make[2]: Entering directory `/.../.source/linux.perf/tools/perf'
make[2]: *** No rule to make target `PERF-VERSION-FILE'. Stop.

Signed-off-by: Robert Richter <[email protected]>
Cc: Ingo Molnar <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
tools/perf/Documentation/Makefile | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tools/perf/Documentation/Makefile b/tools/perf/Documentation/Makefile
index ca600e0..9f2e44f 100644
--- a/tools/perf/Documentation/Makefile
+++ b/tools/perf/Documentation/Makefile
@@ -195,10 +195,10 @@ install-pdf: pdf
#install-html: html
# '$(SHELL_PATH_SQ)' ./install-webdoc.sh $(DESTDIR)$(htmldir)

-../PERF-VERSION-FILE: .FORCE-PERF-VERSION-FILE
- $(QUIET_SUBDIR0)../ $(QUIET_SUBDIR1) PERF-VERSION-FILE
+$(OUTPUT)PERF-VERSION-FILE: .FORCE-PERF-VERSION-FILE
+ $(QUIET_SUBDIR0)../ $(QUIET_SUBDIR1) $(OUTPUT)PERF-VERSION-FILE

--include ../PERF-VERSION-FILE
+-include $(OUTPUT)PERF-VERSION-FILE

#
# Determine "include::" file references in asciidoc files.
--
1.7.9.2.358.g22243

2012-08-08 17:17:08

by Arnaldo Carvalho de Melo

Subject: [PATCH 06/11] perf list: Document precise event sampling for AMD IBS

From: Robert Richter <[email protected]>

Updating man perf-list.

Signed-off-by: Robert Richter <[email protected]>
Cc: Ingo Molnar <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
tools/perf/Documentation/perf-list.txt | 42 +++++++++++++++++++++++---------
1 file changed, 30 insertions(+), 12 deletions(-)

diff --git a/tools/perf/Documentation/perf-list.txt b/tools/perf/Documentation/perf-list.txt
index 232be51..d1e39dc 100644
--- a/tools/perf/Documentation/perf-list.txt
+++ b/tools/perf/Documentation/perf-list.txt
@@ -20,20 +20,38 @@ EVENT MODIFIERS
---------------

Events can optionally have a modifer by appending a colon and one or
-more modifiers. Modifiers allow the user to restrict when events are
-counted with 'u' for user-space, 'k' for kernel, 'h' for hypervisor.
-Additional modifiers are 'G' for guest counting (in KVM guests) and 'H'
-for host counting (not in KVM guests).
+more modifiers. Modifiers allow the user to restrict the events to be
+counted. The following modifiers exist:
+
+ u - user-space counting
+ k - kernel counting
+ h - hypervisor counting
+ G - guest counting (in KVM guests)
+ H - host counting (not in KVM guests)
+ p - precise level

The 'p' modifier can be used for specifying how precise the instruction
-address should be. The 'p' modifier is currently only implemented for
-Intel PEBS and can be specified multiple times:
- 0 - SAMPLE_IP can have arbitrary skid
- 1 - SAMPLE_IP must have constant skid
- 2 - SAMPLE_IP requested to have 0 skid
- 3 - SAMPLE_IP must have 0 skid
+address should be. The 'p' modifier can be specified multiple times:
+
+ 0 - SAMPLE_IP can have arbitrary skid
+ 1 - SAMPLE_IP must have constant skid
+ 2 - SAMPLE_IP requested to have 0 skid
+ 3 - SAMPLE_IP must have 0 skid
+
+For Intel systems precise event sampling is implemented with PEBS
+which supports up to precise-level 2.
+
+On AMD systems it is implemented using IBS (up to precise-level 2).
+The precise modifier works with event types 0x76 (cpu-cycles, CPU
+clocks not halted) and 0xC1 (micro-ops retired). Both events map to
+IBS execution sampling (IBS op) with the IBS Op Counter Control bit
+(IbsOpCntCtl) set respectively (see AMD64 Architecture Programmer’s
+Manual Volume 2: System Programming, 13.3 Instruction-Based
+Sampling). Examples to use IBS:

-The PEBS implementation now supports up to 2.
+ perf record -a -e cpu-cycles:p ... # use ibs op counting cycles
+ perf record -a -e r076:p ... # same as -e cpu-cycles:p
+ perf record -a -e r0C1:p ... # use ibs op counting micro-ops

RAW HARDWARE EVENT DESCRIPTOR
-----------------------------
@@ -97,4 +115,4 @@ SEE ALSO
linkperf:perf-stat[1], linkperf:perf-top[1],
linkperf:perf-record[1],
http://www.intel.com/Assets/PDF/manual/253669.pdf[Intel® 64 and IA-32 Architectures Software Developer's Manual Volume 3B: System Programming Guide],
-http://support.amd.com/us/Processor_TechDocs/24593.pdf[AMD64 Architecture Programmer’s Manual Volume 2: System Programming]
+http://support.amd.com/us/Processor_TechDocs/24593_APM_v2.pdf[AMD64 Architecture Programmer’s Manual Volume 2: System Programming]
--
1.7.9.2.358.g22243

2012-08-09 03:08:12

by Namhyung Kim

Subject: Re: [PATCH 11/11] perf scripts python: Add event_analyzing_sample.py as a sample for general event handling

Hi Arnaldo and Ingo,

On Wed, 8 Aug 2012 14:13:48 -0300, Arnaldo Carvalho de Melo wrote:
> From: Feng Tang <[email protected]>
>
> Currently only trace point events are supported in perf/python script,
> the first 3 patches of this serie add the support for all types of
> events. This script is just a simple sample to show how to gather the
> basic information of the events and analyze them.
>
> This script will create one object for each event sample and insert them
> into a table in a database, then leverage the simple SQL commands to
> sort/group them. User can modify or write their brand new functions
> according to their specific requirment.
>
> Here is the sample of how to use the script:
>
> $ perf record -a tree
> $ perf script -s process_event.py

Please edit the script name to event_analyzing_sample.py at least to
prevent future confusion. For other issues, please see my review
comments on Feng's original posts. (They can be incremental.)

>
> There is 100 records in gen_events table
> Statistics about the general events grouped by thread/symbol/dso:
>
> comm number histgram
> ==========================================
> swapper 56 ######
> tree 20 #####
> perf 10 ####
> sshd 8 ####
> kworker/7:2 4 ###
> ksoftirqd/7 1 #
> plugin-containe 1 #
>
> symbol number histgram
> ==========================================================
> native_write_msr_safe 40 ######
> __lock_acquire 8 ####
> ftrace_graph_caller 4 ###
> prepare_ftrace_return 4 ###
> intel_idle 3 ##
> native_sched_clock 3 ##
> Unknown_symbol 2 ##
> do_softirq 2 ##
> lock_release 2 ##
> lock_release_holdtime 2 ##
> trace_graph_entry 2 ##
> _IO_putc 1 #
> __d_lookup_rcu 1 #
> __do_fault 1 #
> __schedule 1 #
> _raw_spin_lock 1 #
> delay_tsc 1 #
> generic_exec_single 1 #
> generic_fillattr 1 #
>
> dso number histgram
> ==================================================================
> [kernel.kallsyms] 95 #######
> /lib/libc-2.12.1.so 5 ###
>
> Signed-off-by: Feng Tang <[email protected]>
> Cc: Andi Kleen <[email protected]>
> Cc: David Ahern <[email protected]>
> Cc: Ingo Molnar <[email protected]>
> Cc: Peter Zijlstra <[email protected]>
> Cc: Robert Richter <[email protected]>
> Cc: Stephane Eranian <[email protected]>
> Link: http://lkml.kernel.org/r/[email protected]
> Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
> ---
> .../perf/scripts/python/event_analyzing_sample.py | 193 ++++++++++++++++++++
> 1 file changed, 193 insertions(+)
> create mode 100644 tools/perf/scripts/python/event_analyzing_sample.py
>
> diff --git a/tools/perf/scripts/python/event_analyzing_sample.py b/tools/perf/scripts/python/event_analyzing_sample.py
> new file mode 100644
> index 0000000..46f05aa
> --- /dev/null
> +++ b/tools/perf/scripts/python/event_analyzing_sample.py
> @@ -0,0 +1,193 @@
> +# process_event.py: general event handler in python

Hopefully here also.

Thanks,
Namhyung


> +#
> +# Current perf report is alreay very powerful with the anotation integrated,
> +# and this script is not trying to be as powerful as perf report, but
> +# providing end user/developer a flexible way to analyze the events other
> +# than trace points.
> +#
> +# The 2 database related functions in this script just show how to gather
> +# the basic information, and users can modify and write their own functions
> +# according to their specific requirment.
> +#
> +# The first sample "show_general_events" just does a baisc grouping for all
> +# generic events with the help of sqlite, and the 2nd one "show_pebs_ll" is
> +# for a x86 HW PMU event: PEBS with load latency data.
> +#
> +
> +import os
> +import sys
> +import math
> +import struct
> +import sqlite3
> +
> +sys.path.append(os.environ['PERF_EXEC_PATH'] + \
> + '/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
> +
> +from perf_trace_context import *
> +from EventClass import *
> +
> +#
> +# If the perf.data has a big number of samples, then the insert operation
> +# will be very time consuming (about 10+ minutes for 10000 samples) if the
> +# .db database is on disk. Move the .db file to RAM based FS to speedup
> +# the handling, which will cut the time down to several seconds.
> +#
> +con = sqlite3.connect("/dev/shm/perf.db")
> +con.isolation_level = None
> +
> +def trace_begin():
> + print "In trace_begin:\n"
> +
> + #
> + # Will create several tables at the start, pebs_ll is for PEBS data with
> + # load latency info, while gen_events is for general event.
> + #
> + con.execute("""
> + create table if not exists gen_events (
> + name text,
> + symbol text,
> + comm text,
> + dso text
> + );""")
> + con.execute("""
> + create table if not exists pebs_ll (
> + name text,
> + symbol text,
> + comm text,
> + dso text,
> + flags integer,
> + ip integer,
> + status integer,
> + dse integer,
> + dla integer,
> + lat integer
> + );""")
> +
> +#
> +# Create and insert event object to a database so that user could
> +# do more analysis with simple database commands.
> +#
> +def process_event(param_dict):
> + event_attr = param_dict["attr"]
> + sample = param_dict["sample"]
> + raw_buf = param_dict["raw_buf"]
> + comm = param_dict["comm"]
> + name = param_dict["ev_name"]
> +
> + # Symbol and dso info are not always resolved
> + if (param_dict.has_key("dso")):
> + dso = param_dict["dso"]
> + else:
> + dso = "Unknown_dso"
> +
> + if (param_dict.has_key("symbol")):
> + symbol = param_dict["symbol"]
> + else:
> + symbol = "Unknown_symbol"
> +
> + # Creat the event object and insert it to the right table in database
> + event = create_event(name, comm, dso, symbol, raw_buf)
> + insert_db(event)
> +
> +def insert_db(event):
> + if event.ev_type == EVTYPE_GENERIC:
> + con.execute("insert into gen_events values(?, ?, ?, ?)",
> + (event.name, event.symbol, event.comm, event.dso))
> + elif event.ev_type == EVTYPE_PEBS_LL:
> + event.ip &= 0x7fffffffffffffff
> + event.dla &= 0x7fffffffffffffff
> + con.execute("insert into pebs_ll values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
> + (event.name, event.symbol, event.comm, event.dso, event.flags,
> + event.ip, event.status, event.dse, event.dla, event.lat))
> +
> +def trace_end():
> + print "In trace_end:\n"
> + # We show the basic info for the 2 type of event classes
> + show_general_events()
> + show_pebs_ll()
> + con.close()
> +
> +#
> +# As the event number may be very big, so we can't use linear way
> +# to show the histgram in real number, but use a log2 algorithm.
> +#
> +
> +def num2sym(num):
> + # Each number will have at least one '#'
> + snum = '#' * (int)(math.log(num, 2) + 1)
> + return snum
> +
> +def show_general_events():
> +
> + # Check the total record number in the table
> + count = con.execute("select count(*) from gen_events")
> + for t in count:
> + print "There is %d records in gen_events table" % t[0]
> + if t[0] == 0:
> + return
> +
> + print "Statistics about the general events grouped by thread/symbol/dso: \n"
> +
> + # Group by thread
> + commq = con.execute("select comm, count(comm) from gen_events group by comm order by -count(comm)")
> + print "\n%16s %8s %16s\n%s" % ("comm", "number", "histgram", "="*42)
> + for row in commq:
> + print "%16s %8d %s" % (row[0], row[1], num2sym(row[1]))
> +
> + # Group by symbol
> + print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histgram", "="*58)
> + symbolq = con.execute("select symbol, count(symbol) from gen_events group by symbol order by -count(symbol)")
> + for row in symbolq:
> + print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))
> +
> + # Group by dso
> + print "\n%40s %8s %16s\n%s" % ("dso", "number", "histgram", "="*74)
> + dsoq = con.execute("select dso, count(dso) from gen_events group by dso order by -count(dso)")
> + for row in dsoq:
> + print "%40s %8d %s" % (row[0], row[1], num2sym(row[1]))
> +
> +#
> +# This function just shows the basic info, and we could do more with the
> +# data in the tables, like checking the function parameters when some
> +# big latency events happen.
> +#
> +def show_pebs_ll():
> +
> + count = con.execute("select count(*) from pebs_ll")
> + for t in count:
> + print "There is %d records in pebs_ll table" % t[0]
> + if t[0] == 0:
> + return
> +
> + print "Statistics about the PEBS Load Latency events grouped by thread/symbol/dse/latency: \n"
> +
> + # Group by thread
> + commq = con.execute("select comm, count(comm) from pebs_ll group by comm order by -count(comm)")
> + print "\n%16s %8s %16s\n%s" % ("comm", "number", "histgram", "="*42)
> + for row in commq:
> + print "%16s %8d %s" % (row[0], row[1], num2sym(row[1]))
> +
> + # Group by symbol
> + print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histgram", "="*58)
> + symbolq = con.execute("select symbol, count(symbol) from pebs_ll group by symbol order by -count(symbol)")
> + for row in symbolq:
> + print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))
> +
> + # Group by dse
> + dseq = con.execute("select dse, count(dse) from pebs_ll group by dse order by -count(dse)")
> + print "\n%32s %8s %16s\n%s" % ("dse", "number", "histgram", "="*58)
> + for row in dseq:
> + print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))
> +
> + # Group by latency
> + latq = con.execute("select lat, count(lat) from pebs_ll group by lat order by lat")
> + print "\n%32s %8s %16s\n%s" % ("latency", "number", "histgram", "="*58)
> + for row in latq:
> + print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))
> +
> +def trace_unhandled(event_name, context, event_fields_dict):
> + print ' '.join(['%s=%s'%(k,str(v))for k,v in sorted(event_fields_dict.items())])
> +
> +def print_header(event_name, cpu, secs, nsecs, pid, comm):
> + print "%-20s %5u %05u.%09u %8u %-20s " % \
> + (event_name, cpu, secs, nsecs, pid, comm),

2012-08-09 05:29:13

by Feng Tang

Subject: [PATCH] perf script python: Correct handler check and spelling errors

>From 3332f03c6e8f641e26430a25704c364c2f30d833 Mon Sep 17 00:00:00 2001
From: Feng Tang <[email protected]>
Date: Thu, 9 Aug 2012 12:50:56 +0800
Subject: [PATCH] perf script python: Correct handler check and spelling errors

Correct the checking for handler returned by PyDict_GetItemString(), also
fix some spelling error and remove some data code in event_analyzing_sample.py,
as suggested by Namhyung Kim.

Signed-off-by: Feng Tang <[email protected]>
---
.../Perf-Trace-Util/lib/Perf/Trace/EventClass.py | 4 +-
.../perf/scripts/python/event_analyzing_sample.py | 33 ++++++++------------
.../util/scripting-engines/trace-event-python.c | 8 ++---
3 files changed, 18 insertions(+), 27 deletions(-)

diff --git a/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py b/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py
index 6372431..9e09857 100755
--- a/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py
+++ b/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py
@@ -1,12 +1,12 @@
# EventClass.py
#
-# This is a libray defining some events typs classes, which could
+# This is a library defining some events types classes, which could
# be used by other scripts to analyzing the perf samples.
#
# Currently there are just a few classes defined for examples,
# PerfEvent is the base class for all perf event sample, PebsEvent
# is a HW base Intel x86 PEBS event, and user could add more SW/HW
-# event classes based on requriements.
+# event classes based on requirements.

import struct

diff --git a/tools/perf/scripts/python/event_analyzing_sample.py b/tools/perf/scripts/python/event_analyzing_sample.py
index 46f05aa..15555bc 100644
--- a/tools/perf/scripts/python/event_analyzing_sample.py
+++ b/tools/perf/scripts/python/event_analyzing_sample.py
@@ -1,15 +1,15 @@
-# process_event.py: general event handler in python
+# event_analyzing_sample.py: general event handler in python
#
-# Current perf report is alreay very powerful with the anotation integrated,
+# Current perf report is already very powerful with the annotation integrated,
# and this script is not trying to be as powerful as perf report, but
# providing end user/developer a flexible way to analyze the events other
# than trace points.
#
# The 2 database related functions in this script just show how to gather
# the basic information, and users can modify and write their own functions
-# according to their specific requirment.
+# according to their specific requirement.
#
-# The first sample "show_general_events" just does a baisc grouping for all
+# The first function "show_general_events" just does a basic grouping for all
# generic events with the help of sqlite, and the 2nd one "show_pebs_ll" is
# for a x86 HW PMU event: PEBS with load latency data.
#
@@ -85,7 +85,7 @@ def process_event(param_dict):
else:
symbol = "Unknown_symbol"

- # Creat the event object and insert it to the right table in database
+ # Create the event object and insert it to the right table in database
event = create_event(name, comm, dso, symbol, raw_buf)
insert_db(event)

@@ -109,7 +109,7 @@ def trace_end():

#
# As the event number may be very big, so we can't use linear way
-# to show the histgram in real number, but use a log2 algorithm.
+# to show the histogram in real number, but use a log2 algorithm.
#

def num2sym(num):
@@ -130,18 +130,18 @@ def show_general_events():

# Group by thread
commq = con.execute("select comm, count(comm) from gen_events group by comm order by -count(comm)")
- print "\n%16s %8s %16s\n%s" % ("comm", "number", "histgram", "="*42)
+ print "\n%16s %8s %16s\n%s" % ("comm", "number", "histogram", "="*42)
for row in commq:
print "%16s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by symbol
- print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histgram", "="*58)
+ print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histogram", "="*58)
symbolq = con.execute("select symbol, count(symbol) from gen_events group by symbol order by -count(symbol)")
for row in symbolq:
print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by dso
- print "\n%40s %8s %16s\n%s" % ("dso", "number", "histgram", "="*74)
+ print "\n%40s %8s %16s\n%s" % ("dso", "number", "histogram", "="*74)
dsoq = con.execute("select dso, count(dso) from gen_events group by dso order by -count(dso)")
for row in dsoq:
print "%40s %8d %s" % (row[0], row[1], num2sym(row[1]))
@@ -163,31 +163,24 @@ def show_pebs_ll():

# Group by thread
commq = con.execute("select comm, count(comm) from pebs_ll group by comm order by -count(comm)")
- print "\n%16s %8s %16s\n%s" % ("comm", "number", "histgram", "="*42)
+ print "\n%16s %8s %16s\n%s" % ("comm", "number", "histogram", "="*42)
for row in commq:
print "%16s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by symbol
- print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histgram", "="*58)
+ print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histogram", "="*58)
symbolq = con.execute("select symbol, count(symbol) from pebs_ll group by symbol order by -count(symbol)")
for row in symbolq:
print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by dse
dseq = con.execute("select dse, count(dse) from pebs_ll group by dse order by -count(dse)")
- print "\n%32s %8s %16s\n%s" % ("dse", "number", "histgram", "="*58)
+ print "\n%32s %8s %16s\n%s" % ("dse", "number", "histogram", "="*58)
for row in dseq:
print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by latency
latq = con.execute("select lat, count(lat) from pebs_ll group by lat order by lat")
- print "\n%32s %8s %16s\n%s" % ("latency", "number", "histgram", "="*58)
+ print "\n%32s %8s %16s\n%s" % ("latency", "number", "histogram", "="*58)
for row in latq:
print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))
-
-def trace_unhandled(event_name, context, event_fields_dict):
- print ' '.join(['%s=%s'%(k,str(v))for k,v in sorted(event_fields_dict.items())])
-
-def print_header(event_name, cpu, secs, nsecs, pid, comm):
- print "%-20s %5u %05u.%09u %8u %-20s " % \
- (event_name, cpu, secs, nsecs, pid, comm),
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 7e3f576..afba097 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -343,7 +343,7 @@ static void python_process_general_event(union perf_event *perf_event __unused,
struct perf_sample *sample,
struct perf_evsel *evsel,
struct machine *machine __unused,
- struct addr_location *al __unused)
+ struct addr_location *al)
{
PyObject *handler, *retval, *t, *dict;
static char handler_name[64];
@@ -352,7 +352,7 @@ static void python_process_general_event(union perf_event *perf_event __unused,

/*
* Use the MAX_FIELDS to make the function expandable, though
- * currently there is only one itme for the tuple.
+ * currently there is only one item for the tuple.
*/
t = PyTuple_New(MAX_FIELDS);
if (!t)
@@ -365,10 +365,8 @@ static void python_process_general_event(union perf_event *perf_event __unused,
snprintf(handler_name, sizeof(handler_name), "%s", "process_event");

handler = PyDict_GetItemString(main_dict, handler_name);
- if (handler && !PyCallable_Check(handler)) {
- handler = NULL;
+ if (!handler || !PyCallable_Check(handler))
goto exit;
- }

PyDict_SetItemString(dict, "ev_name", PyString_FromString(perf_evsel__name(evsel)));
PyDict_SetItemString(dict, "attr", PyString_FromStringAndSize(
--
1.7.1

2012-08-09 05:41:12

by Namhyung Kim

Subject: Re: [PATCH] perf script python: Correct handler check and spelling errors

On Thu, 9 Aug 2012 13:24:01 +0800, Feng Tang wrote:
> From 3332f03c6e8f641e26430a25704c364c2f30d833 Mon Sep 17 00:00:00 2001
> From: Feng Tang <[email protected]>
> Date: Thu, 9 Aug 2012 12:50:56 +0800
> Subject: [PATCH] perf script python: Correct handler check and spelling errors
>
> Correct the checking for handler returned by PyDict_GetItemString(), also
> fix some spelling error and remove some data code in event_analyzing_sample.py,
> as suggested by Namhyung Kim.
>
> Signed-off-by: Feng Tang <[email protected]>
> ---
[SNIP]
> -
> -def trace_unhandled(event_name, context, event_fields_dict):
> - print ' '.join(['%s=%s'%(k,str(v))for k,v in sorted(event_fields_dict.items())])

I guess we need to have the trace_unhandled as a safety net since it can
be called from the scripting-engine?

Thanks,
Namhyung


> -
> -def print_header(event_name, cpu, secs, nsecs, pid, comm):
> - print "%-20s %5u %05u.%09u %8u %-20s " % \
> - (event_name, cpu, secs, nsecs, pid, comm),
> diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
> index 7e3f576..afba097 100644
> --- a/tools/perf/util/scripting-engines/trace-event-python.c
> +++ b/tools/perf/util/scripting-engines/trace-event-python.c
> @@ -343,7 +343,7 @@ static void python_process_general_event(union perf_event *perf_event __unused,
> struct perf_sample *sample,
> struct perf_evsel *evsel,
> struct machine *machine __unused,
> - struct addr_location *al __unused)
> + struct addr_location *al)
> {
> PyObject *handler, *retval, *t, *dict;
> static char handler_name[64];
> @@ -352,7 +352,7 @@ static void python_process_general_event(union perf_event *perf_event __unused,
>
> /*
> * Use the MAX_FIELDS to make the function expandable, though
> - * currently there is only one itme for the tuple.
> + * currently there is only one item for the tuple.
> */
> t = PyTuple_New(MAX_FIELDS);
> if (!t)
> @@ -365,10 +365,8 @@ static void python_process_general_event(union perf_event *perf_event __unused,
> snprintf(handler_name, sizeof(handler_name), "%s", "process_event");
>
> handler = PyDict_GetItemString(main_dict, handler_name);
> - if (handler && !PyCallable_Check(handler)) {
> - handler = NULL;
> + if (!handler || !PyCallable_Check(handler))
> goto exit;
> - }
>
> PyDict_SetItemString(dict, "ev_name", PyString_FromString(perf_evsel__name(evsel)));
> PyDict_SetItemString(dict, "attr", PyString_FromStringAndSize(

2012-08-09 05:51:25

by Feng Tang

Subject: [PATCH v2] perf script python: Correct handler check and spelling errors

>From efbf00591514aa1d6134a3b940dc627d1a7cc8f8 Mon Sep 17 00:00:00 2001
From: Feng Tang <[email protected]>
Date: Thu, 9 Aug 2012 12:50:56 +0800
Subject: [PATCH] perf script python: Correct handler check and spelling errors

Correct the checking for handler returned by PyDict_GetItemString(), also
fix some spelling error and remove some data code in event_analyzing_sample.py,
as suggested by Namhyung Kim.

v2: restore back the wrongly removed trace_unhandled() func

Signed-off-by: Feng Tang <[email protected]>
---
.../Perf-Trace-Util/lib/Perf/Trace/EventClass.py | 4 +-
.../perf/scripts/python/event_analyzing_sample.py | 30 ++++++++-----------
.../util/scripting-engines/trace-event-python.c | 8 ++---
3 files changed, 18 insertions(+), 24 deletions(-)

diff --git a/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py b/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py
index 6372431..9e09857 100755
--- a/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py
+++ b/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py
@@ -1,12 +1,12 @@
# EventClass.py
#
-# This is a libray defining some events typs classes, which could
+# This is a library defining some events types classes, which could
# be used by other scripts to analyzing the perf samples.
#
# Currently there are just a few classes defined for examples,
# PerfEvent is the base class for all perf event sample, PebsEvent
# is a HW base Intel x86 PEBS event, and user could add more SW/HW
-# event classes based on requriements.
+# event classes based on requirements.

import struct

diff --git a/tools/perf/scripts/python/event_analyzing_sample.py b/tools/perf/scripts/python/event_analyzing_sample.py
index 46f05aa..163c39f 100644
--- a/tools/perf/scripts/python/event_analyzing_sample.py
+++ b/tools/perf/scripts/python/event_analyzing_sample.py
@@ -1,15 +1,15 @@
-# process_event.py: general event handler in python
+# event_analyzing_sample.py: general event handler in python
#
-# Current perf report is alreay very powerful with the anotation integrated,
+# Current perf report is already very powerful with the annotation integrated,
# and this script is not trying to be as powerful as perf report, but
# providing end user/developer a flexible way to analyze the events other
# than trace points.
#
# The 2 database related functions in this script just show how to gather
# the basic information, and users can modify and write their own functions
-# according to their specific requirment.
+# according to their specific requirement.
#
-# The first sample "show_general_events" just does a baisc grouping for all
+# The first function "show_general_events" just does a basic grouping for all
# generic events with the help of sqlite, and the 2nd one "show_pebs_ll" is
# for a x86 HW PMU event: PEBS with load latency data.
#
@@ -85,7 +85,7 @@ def process_event(param_dict):
else:
symbol = "Unknown_symbol"

- # Creat the event object and insert it to the right table in database
+ # Create the event object and insert it to the right table in database
event = create_event(name, comm, dso, symbol, raw_buf)
insert_db(event)

@@ -109,7 +109,7 @@ def trace_end():

#
# As the event number may be very big, so we can't use linear way
-# to show the histgram in real number, but use a log2 algorithm.
+# to show the histogram in real number, but use a log2 algorithm.
#

def num2sym(num):
@@ -130,18 +130,18 @@ def show_general_events():

# Group by thread
commq = con.execute("select comm, count(comm) from gen_events group by comm order by -count(comm)")
- print "\n%16s %8s %16s\n%s" % ("comm", "number", "histgram", "="*42)
+ print "\n%16s %8s %16s\n%s" % ("comm", "number", "histogram", "="*42)
for row in commq:
print "%16s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by symbol
- print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histgram", "="*58)
+ print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histogram", "="*58)
symbolq = con.execute("select symbol, count(symbol) from gen_events group by symbol order by -count(symbol)")
for row in symbolq:
print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by dso
- print "\n%40s %8s %16s\n%s" % ("dso", "number", "histgram", "="*74)
+ print "\n%40s %8s %16s\n%s" % ("dso", "number", "histogram", "="*74)
dsoq = con.execute("select dso, count(dso) from gen_events group by dso order by -count(dso)")
for row in dsoq:
print "%40s %8d %s" % (row[0], row[1], num2sym(row[1]))
@@ -163,31 +163,27 @@ def show_pebs_ll():

# Group by thread
commq = con.execute("select comm, count(comm) from pebs_ll group by comm order by -count(comm)")
- print "\n%16s %8s %16s\n%s" % ("comm", "number", "histgram", "="*42)
+ print "\n%16s %8s %16s\n%s" % ("comm", "number", "histogram", "="*42)
for row in commq:
print "%16s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by symbol
- print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histgram", "="*58)
+ print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histogram", "="*58)
symbolq = con.execute("select symbol, count(symbol) from pebs_ll group by symbol order by -count(symbol)")
for row in symbolq:
print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by dse
dseq = con.execute("select dse, count(dse) from pebs_ll group by dse order by -count(dse)")
- print "\n%32s %8s %16s\n%s" % ("dse", "number", "histgram", "="*58)
+ print "\n%32s %8s %16s\n%s" % ("dse", "number", "histogram", "="*58)
for row in dseq:
print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by latency
latq = con.execute("select lat, count(lat) from pebs_ll group by lat order by lat")
- print "\n%32s %8s %16s\n%s" % ("latency", "number", "histgram", "="*58)
+ print "\n%32s %8s %16s\n%s" % ("latency", "number", "histogram", "="*58)
for row in latq:
print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))

def trace_unhandled(event_name, context, event_fields_dict):
print ' '.join(['%s=%s'%(k,str(v))for k,v in sorted(event_fields_dict.items())])
-
-def print_header(event_name, cpu, secs, nsecs, pid, comm):
- print "%-20s %5u %05u.%09u %8u %-20s " % \
- (event_name, cpu, secs, nsecs, pid, comm),
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 7e3f576..afba097 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -343,7 +343,7 @@ static void python_process_general_event(union perf_event *perf_event __unused,
struct perf_sample *sample,
struct perf_evsel *evsel,
struct machine *machine __unused,
- struct addr_location *al __unused)
+ struct addr_location *al)
{
PyObject *handler, *retval, *t, *dict;
static char handler_name[64];
@@ -352,7 +352,7 @@ static void python_process_general_event(union perf_event *perf_event __unused,

/*
* Use the MAX_FIELDS to make the function expandable, though
- * currently there is only one itme for the tuple.
+ * currently there is only one item for the tuple.
*/
t = PyTuple_New(MAX_FIELDS);
if (!t)
@@ -365,10 +365,8 @@ static void python_process_general_event(union perf_event *perf_event __unused,
snprintf(handler_name, sizeof(handler_name), "%s", "process_event");

handler = PyDict_GetItemString(main_dict, handler_name);
- if (handler && !PyCallable_Check(handler)) {
- handler = NULL;
+ if (!handler || !PyCallable_Check(handler))
goto exit;
- }

PyDict_SetItemString(dict, "ev_name", PyString_FromString(perf_evsel__name(evsel)));
PyDict_SetItemString(dict, "attr", PyString_FromStringAndSize(
--
1.7.1

2012-08-09 06:21:39

by Namhyung Kim

Subject: Re: [PATCH v2] perf script python: Correct handler check and spelling errors

On Thu, 9 Aug 2012 13:46:13 +0800, Feng Tang wrote:
> From efbf00591514aa1d6134a3b940dc627d1a7cc8f8 Mon Sep 17 00:00:00 2001
> From: Feng Tang <[email protected]>
> Date: Thu, 9 Aug 2012 12:50:56 +0800
> Subject: [PATCH] perf script python: Correct handler check and spelling errors
>
> Correct the checking for handler returned by PyDict_GetItemString(), also
> fix some spelling error and remove some data code in event_analyzing_sample.py,
> as suggested by Namhyung Kim.
>
> v2: restore back the wrongly removed trace_unhandled() func
>
> Signed-off-by: Feng Tang <[email protected]>

Acked-by: Namhyung Kim <[email protected]>

Thanks!

2012-08-21 15:37:33

by Feng Tang

Subject: [tip:perf/core] perf script python: Correct handler check and spelling errors

Commit-ID: 87b6a3ad40ba304ec468b972e979e7e410852476
Gitweb: http://git.kernel.org/tip/87b6a3ad40ba304ec468b972e979e7e410852476
Author: Feng Tang <[email protected]>
AuthorDate: Thu, 9 Aug 2012 13:46:13 +0800
Committer: Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Thu, 9 Aug 2012 13:26:19 -0300

perf script python: Correct handler check and spelling errors

Correct the checking for handler returned by PyDict_GetItemString(),
also fix some spelling error and remove some data code in
event_analyzing_sample.py, as suggested by Namhyung Kim.

v2: restore back the wrongly removed trace_unhandled() func

Signed-off-by: Feng Tang <[email protected]>
Acked-by: Namhyung Kim <[email protected]>
Cc: Andi Kleen <[email protected]>
Cc: David Ahern <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Robert Richter <[email protected]>
Cc: Stephane Eranian <[email protected]>
Link: http://lkml.kernel.org/r/20120809134613.067104c4@feng-i7
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
.../Perf-Trace-Util/lib/Perf/Trace/EventClass.py | 4 +-
.../perf/scripts/python/event_analyzing_sample.py | 30 ++++++++-----------
.../util/scripting-engines/trace-event-python.c | 8 ++---
3 files changed, 18 insertions(+), 24 deletions(-)

diff --git a/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py b/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py
index 6372431..9e098579 100755
--- a/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py
+++ b/tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py
@@ -1,12 +1,12 @@
# EventClass.py
#
-# This is a libray defining some events typs classes, which could
+# This is a library defining some events types classes, which could
# be used by other scripts to analyzing the perf samples.
#
# Currently there are just a few classes defined for examples,
# PerfEvent is the base class for all perf event sample, PebsEvent
# is a HW base Intel x86 PEBS event, and user could add more SW/HW
-# event classes based on requriements.
+# event classes based on requirements.

import struct

diff --git a/tools/perf/scripts/python/event_analyzing_sample.py b/tools/perf/scripts/python/event_analyzing_sample.py
index 46f05aa..163c39f 100644
--- a/tools/perf/scripts/python/event_analyzing_sample.py
+++ b/tools/perf/scripts/python/event_analyzing_sample.py
@@ -1,15 +1,15 @@
-# process_event.py: general event handler in python
+# event_analyzing_sample.py: general event handler in python
#
-# Current perf report is alreay very powerful with the anotation integrated,
+# Current perf report is already very powerful with the annotation integrated,
# and this script is not trying to be as powerful as perf report, but
# providing end user/developer a flexible way to analyze the events other
# than trace points.
#
# The 2 database related functions in this script just show how to gather
# the basic information, and users can modify and write their own functions
-# according to their specific requirment.
+# according to their specific requirement.
#
-# The first sample "show_general_events" just does a baisc grouping for all
+# The first function "show_general_events" just does a basic grouping for all
# generic events with the help of sqlite, and the 2nd one "show_pebs_ll" is
# for a x86 HW PMU event: PEBS with load latency data.
#
@@ -85,7 +85,7 @@ def process_event(param_dict):
else:
symbol = "Unknown_symbol"

- # Creat the event object and insert it to the right table in database
+ # Create the event object and insert it to the right table in database
event = create_event(name, comm, dso, symbol, raw_buf)
insert_db(event)

@@ -109,7 +109,7 @@ def trace_end():

#
# As the event number may be very big, so we can't use linear way
-# to show the histgram in real number, but use a log2 algorithm.
+# to show the histogram in real number, but use a log2 algorithm.
#

def num2sym(num):
@@ -130,18 +130,18 @@ def show_general_events():

# Group by thread
commq = con.execute("select comm, count(comm) from gen_events group by comm order by -count(comm)")
- print "\n%16s %8s %16s\n%s" % ("comm", "number", "histgram", "="*42)
+ print "\n%16s %8s %16s\n%s" % ("comm", "number", "histogram", "="*42)
for row in commq:
print "%16s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by symbol
- print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histgram", "="*58)
+ print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histogram", "="*58)
symbolq = con.execute("select symbol, count(symbol) from gen_events group by symbol order by -count(symbol)")
for row in symbolq:
print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by dso
- print "\n%40s %8s %16s\n%s" % ("dso", "number", "histgram", "="*74)
+ print "\n%40s %8s %16s\n%s" % ("dso", "number", "histogram", "="*74)
dsoq = con.execute("select dso, count(dso) from gen_events group by dso order by -count(dso)")
for row in dsoq:
print "%40s %8d %s" % (row[0], row[1], num2sym(row[1]))
@@ -163,31 +163,27 @@ def show_pebs_ll():

# Group by thread
commq = con.execute("select comm, count(comm) from pebs_ll group by comm order by -count(comm)")
- print "\n%16s %8s %16s\n%s" % ("comm", "number", "histgram", "="*42)
+ print "\n%16s %8s %16s\n%s" % ("comm", "number", "histogram", "="*42)
for row in commq:
print "%16s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by symbol
- print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histgram", "="*58)
+ print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histogram", "="*58)
symbolq = con.execute("select symbol, count(symbol) from pebs_ll group by symbol order by -count(symbol)")
for row in symbolq:
print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by dse
dseq = con.execute("select dse, count(dse) from pebs_ll group by dse order by -count(dse)")
- print "\n%32s %8s %16s\n%s" % ("dse", "number", "histgram", "="*58)
+ print "\n%32s %8s %16s\n%s" % ("dse", "number", "histogram", "="*58)
for row in dseq:
print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))

# Group by latency
latq = con.execute("select lat, count(lat) from pebs_ll group by lat order by lat")
- print "\n%32s %8s %16s\n%s" % ("latency", "number", "histgram", "="*58)
+ print "\n%32s %8s %16s\n%s" % ("latency", "number", "histogram", "="*58)
for row in latq:
print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))

def trace_unhandled(event_name, context, event_fields_dict):
print ' '.join(['%s=%s'%(k,str(v))for k,v in sorted(event_fields_dict.items())])
-
-def print_header(event_name, cpu, secs, nsecs, pid, comm):
- print "%-20s %5u %05u.%09u %8u %-20s " % \
- (event_name, cpu, secs, nsecs, pid, comm),
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 7e3f576..afba097 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -343,7 +343,7 @@ static void python_process_general_event(union perf_event *perf_event __unused,
struct perf_sample *sample,
struct perf_evsel *evsel,
struct machine *machine __unused,
- struct addr_location *al __unused)
+ struct addr_location *al)
{
PyObject *handler, *retval, *t, *dict;
static char handler_name[64];
@@ -352,7 +352,7 @@ static void python_process_general_event(union perf_event *perf_event __unused,

/*
* Use the MAX_FIELDS to make the function expandable, though
- * currently there is only one itme for the tuple.
+ * currently there is only one item for the tuple.
*/
t = PyTuple_New(MAX_FIELDS);
if (!t)
@@ -365,10 +365,8 @@ static void python_process_general_event(union perf_event *perf_event __unused,
snprintf(handler_name, sizeof(handler_name), "%s", "process_event");

handler = PyDict_GetItemString(main_dict, handler_name);
- if (handler && !PyCallable_Check(handler)) {
- handler = NULL;
+ if (!handler || !PyCallable_Check(handler))
goto exit;
- }

PyDict_SetItemString(dict, "ev_name", PyString_FromString(perf_evsel__name(evsel)));
PyDict_SetItemString(dict, "attr", PyString_FromStringAndSize(