2011-06-28 07:41:14

by Akihiro Nagai

[permalink] [raw]
Subject: [PATCH -tip 0/4] perf script: add BTS analysis features

Hi Frederic, David,

This patch series adds the functions to analyze BTS logs to perf-script and,
makes perf-script more informative.
The patches add the following functions.
- Resolve DSOs and symbols for BTS's branch_from addresses
- Resolve DSOs and symbols for user-space address
- Show the offset of symbols with '--show-symbol-offset' option.
- Resolve the real path of [kernel.kallsym] using
'--show-kernel-path' option.

Usage:
First, we get the BTS log with the following command.
# perf record -e branches:u -c 1 -d <command>

Second, analyze it.
# perf script -f ip,addr,sym,dso [--show-symbol-offset] [--show-kernel-path]
This command's output format is:
<branch_to addr> <branch_to function(+offset)> <branch_to DSO> <branch_from addr> <branch_from function(+offset)> <branch_from DSO>

Output sample:
# perf record -e branches:u -c 1 -d ls
[snip]
# perf script -f ip,addr,sym,dso --show-symbol-offset --show-kernel-path
301ec016b0 (/lib/modules/3.0.0-rc3-tip+/build/vmlinux) ffffffff81467612 irq_return+0x0 (/lib/modules/3.0.0-rc3-tip+/build/vmlinux)
301ec016b0 _start+0x0 (/lib64/ld-2.14.so) ffffffff81467612 irq_return+0x0 (/lib/modules/3.0.0-rc3-tip+/build/vmlinux)
301ec04b70 _dl_start+0x0 (/lib64/ld-2.14.so) 301ec016b3 _start+0x3 (/lib64/ld-2.14.so)
301ec04b70 _dl_start+0x0 (/lib64/ld-2.14.so) ffffffff81467612 irq_return+0x0 (/lib/modules/3.0.0-rc3-tip+/build/vmlinux)
301ec04b96 _dl_start+0x26 (/lib64/ld-2.14.so) ffffffff81467612 irq_return+0x0 (/lib/modules/3.0.0-rc3-tip+/build/vmlinux)
301ec04b9d _dl_start+0x2d (/lib64/ld-2.14.so) ffffffff81467612 irq_return+0x0 (/lib/modules/3.0.0-rc3-tip+/build/vmlinux)
301ec04c0d _dl_start+0x9d (/lib64/ld-2.14.so) 301ec04beb _dl_start+0x7b (/lib64/ld-2.14.so)
[snip]
401fd0 main+0x0 (/root/bin/ls) 301f021399 __libc_start_main+0xe9 (/lib64/libc-2.14.so)
409ad0 set_program_name+0x0 (/root/bin/ls) 401ff3 main+0x23 (/root/bin/ls)
409ad0 set_program_name+0x0 (/root/bin/ls) ffffffff81467612 irq_return+0x0 (/lib/modules/3.0.0-rc3-tip+/build/vmlinux)
401ca8 strrchr@plt+0x0 (/root/bin/ls) 409ade set_program_name+0xe (/root/bin/ls)
401cae strrchr@plt+0x6 (/root/bin/ls) 401ca8 strrchr@plt+0x0 (/root/bin/ls)
401a38 _init+0x18 (/root/bin/ls) 401cb3 strrchr@plt+0xb (/root/bin/ls)
301ec13840 _dl_runtime_resolve+0x0 (/lib64/ld-2.14.so) 401a3e _init+0x1e (/root/bin/ls)
301ec0d6a0 _dl_fixup+0x0 (/lib64/ld-2.14.so) 301ec13870 _dl_runtime_resolve+0x30 (/lib64/ld-2.14.so)
[snip]

TODO:
- add source code path field, line number field ...etc.
- add record/report script to use easily and, show human-friendly output.
- filtering kernel functions using scripts


BTW, I couldn't run a script generated by 'perf script -g {pyton|perl}'
with branch events. How do I make scripts analyzing branch events?

# perf record -e branches:u -c 1 -d ls
# perf script -g python
# perf script -s perf-script.py
in trace_begin
Fatal: no event_list!
# perf script -g perl
# perf script -s perf-script.pl
Fatal: no event_list!

Thanks,

---

Akihiro Nagai (4):
perf script: add option resolving vmlinux path
perf script: add the option to show the offset of symbols
perf script: print DSOs and symbols for BTS branch_from addr
perf script: resolve DSOs and symbols for user-space


tools/perf/Documentation/perf-script.txt | 6 ++++
tools/perf/builtin-script.c | 44 +++++++++++++++++++-----------
tools/perf/util/event.c | 4 +++
tools/perf/util/session.c | 44 ++++++++++++++----------------
tools/perf/util/session.h | 3 +-
tools/perf/util/symbol.c | 12 ++++++++
tools/perf/util/symbol.h | 2 +
7 files changed, 74 insertions(+), 41 deletions(-)

--
Akihiro Nagai ([email protected])


2011-06-28 07:41:18

by Akihiro Nagai

[permalink] [raw]
Subject: [PATCH -tip 1/4] perf script: resolve DSOs and symbols for user-space

Resolve user-space DSOs and symbols.
Latest perf-script can resolve DSOs and symbols for only kernel
and kernel modules. This patch resolves them for other
executable binaries.

# perf script -f ip,addr,sym,dso
7f87279646b0 ([kernel.kallsyms]) ffffffff81467612 irq_return ([kernel.kallsyms])
7f87279646b0 _start (/lib64/ld-2.14.so) ffffffff81467612 irq_return ([kernel.kallsyms])
7f8727967b70 _dl_start (/lib64/ld-2.14.so) 7f87279646b3 _start (/lib64/ld-2.14.so)
7f8727967b70 _dl_start (/lib64/ld-2.14.so) ffffffff81467612 irq_return ([kernel.kallsyms])
[snip]

Signed-off-by: Akihiro Nagai <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Frederic Weisbecker <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: David Ahern <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
---

tools/perf/builtin-script.c | 3 +++
tools/perf/util/event.c | 4 ++++
tools/perf/util/session.c | 2 +-
3 files changed, 8 insertions(+), 1 deletions(-)

diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 3056b45..e45433f 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -327,6 +327,9 @@ static void print_sample_addr(union perf_event *event,
thread__find_addr_map(thread, session, cpumode, MAP__FUNCTION,
event->ip.pid, sample->addr, &al);
if (!al.map)
+ thread__find_addr_map(thread, session, PERF_RECORD_MISC_USER,
+ MAP__FUNCTION, event->ip.pid, sample->addr, &al);
+ if (!al.map)
thread__find_addr_map(thread, session, cpumode, MAP__VARIABLE,
event->ip.pid, sample->addr, &al);

diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 3c1b8a6..68e7af9 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -771,6 +771,10 @@ int perf_event__preprocess_sample(const union perf_event *event,

thread__find_addr_map(thread, session, cpumode, MAP__FUNCTION,
event->ip.pid, event->ip.ip, al);
+ if (!al->map)
+ thread__find_addr_map(thread, session, PERF_RECORD_MISC_USER,
+ MAP__FUNCTION, event->ip.pid, event->ip.ip, al);
+
dump_printf(" ...... dso: %s\n",
al->map ? al->map->dso->long_name :
al->level == 'H' ? "[hypervisor]" : "<not found>");
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index b723f21..632266f 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1257,7 +1257,7 @@ void perf_session__print_ip(union perf_event *event,
}

} else {
- printf("%16" PRIx64, al.addr);
+ printf("%16" PRIx64, sample->ip);
if (print_sym) {
if (al.sym && al.sym->name)
symname = al.sym->name;

2011-06-28 07:41:23

by Akihiro Nagai

[permalink] [raw]
Subject: [PATCH -tip 2/4] perf script: print DSOs and symbols for BTS branch_from addr

Print DSOs and symbols for branch_from address of BTS.
BTS records branch_from address in 'addr' fields,
and branch_to address in 'ip' field. Latest perf-script
resolves DSOs and symbols only for 'ip' field.
This patch resolves them for 'addr' field too.

# perf script -f ip,addr,dso,sym
3f03e016b0 ([kernel.kallsyms]) ffffffff81467612 irq_return ([kernel.kallsyms])
3f03e016b0 _start (/lib64/ld-2.14.so) ffffffff81467612 irq_return ([kernel.kallsyms])
3f03e04b80 _dl_start (/lib64/ld-2.14.so) 3f03e016b3 _start (/lib64/ld-2.14.so)
3f03e04b80 _dl_start (/lib64/ld-2.14.so) ffffffff81467612 irq_return ([kernel.kallsyms])
3f03e04ba6 _dl_start (/lib64/ld-2.14.so) ffffffff81467612 irq_return ([kernel.kallsyms])

Signed-off-by: Akihiro Nagai <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Frederic Weisbecker <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: David Ahern <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
---

tools/perf/builtin-script.c | 6 ++++++
1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index e45433f..2d68086 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -306,6 +306,12 @@ static bool sample_addr_correlates_sym(struct perf_event_attr *attr)
(attr->config == PERF_COUNT_SW_PAGE_FAULTS_MAJ)))
return true;

+ /* BTS Events */
+ if ((attr->type == PERF_TYPE_HARDWARE) &&
+ (attr->config & PERF_COUNT_HW_BRANCH_INSTRUCTIONS) &&
+ (attr->sample_period == 1))
+ return true;
+
return false;
}

2011-06-28 07:41:28

by Akihiro Nagai

[permalink] [raw]
Subject: [PATCH -tip 3/4] perf script: add the option to show the offset of symbols

Add the option '--show-symbol-offset' to show the offset from
the symbols in the output of perf-script. We can get the more
detailed address information.

Output sample:
# perf script -f ip,addr,sym --show-symbol-offset
301ec016b0 ffffffff81467612 irq_return+0x0
301ec016b0 _start+0x0 ffffffff81467612 irq_return+0x0
301ec04b70 _dl_start+0x0 301ec016b3 _start+0x3
301ec04b70 _dl_start+0x0 ffffffff81467612 irq_return+0x0
301ec04b96 _dl_start+0x26 ffffffff81467612 irq_return+0x0
301ec04b9d _dl_start+0x2d ffffffff81467612 irq_return+0x0
301ec04c0d _dl_start+0x9d 301ec04beb _dl_start+0x7b
301ec04bf0 _dl_start+0x80 301ec04c11 _dl_start+0xa1
[snip]

Signed-off-by: Akihiro Nagai <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Frederic Weisbecker <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: David Ahern <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
---

tools/perf/Documentation/perf-script.txt | 3 +++
tools/perf/builtin-script.c | 20 ++++++++++++++------
tools/perf/util/session.c | 16 ++++++++++------
tools/perf/util/session.h | 3 ++-
4 files changed, 29 insertions(+), 13 deletions(-)

diff --git a/tools/perf/Documentation/perf-script.txt b/tools/perf/Documentation/perf-script.txt
index c6068cb..39c4721 100644
--- a/tools/perf/Documentation/perf-script.txt
+++ b/tools/perf/Documentation/perf-script.txt
@@ -182,6 +182,9 @@ OPTIONS
--hide-call-graph::
When printing symbols do not display call chain.

+--show-symbol-offset::
+ Show the offset of the symbols.
+
SEE ALSO
--------
linkperf:perf-record[1], linkperf:perf-script-perl[1],
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 2d68086..f9b7047 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -21,6 +21,7 @@ static u64 last_timestamp;
static u64 nr_unordered;
extern const struct option record_options[];
static bool no_callchain;
+static bool show_symbol_offset;

enum perf_output_field {
PERF_OUTPUT_COMM = 1U << 0,
@@ -324,6 +325,7 @@ static void print_sample_addr(union perf_event *event,
struct addr_location al;
u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
const char *symname, *dsoname;
+ unsigned long offset;

printf("%16" PRIx64, sample->addr);

@@ -346,12 +348,15 @@ static void print_sample_addr(union perf_event *event,
al.sym = map__find_symbol(al.map, al.addr, NULL);

if (PRINT_FIELD(SYM)) {
- if (al.sym && al.sym->name)
+ printf(" ");
+ if (al.sym && al.sym->name) {
symname = al.sym->name;
- else
- symname = "";
-
- printf(" %16s", symname);
+ offset = al.addr - al.sym->start;
+ if (show_symbol_offset)
+ printf("%16s+0x%lx", symname, offset);
+ else
+ printf("%16s", symname);
+ }
}

if (PRINT_FIELD(DSO)) {
@@ -390,7 +395,8 @@ static void process_event(union perf_event *event __unused,
else
printf("\n");
perf_session__print_ip(event, sample, session,
- PRINT_FIELD(SYM), PRINT_FIELD(DSO));
+ PRINT_FIELD(SYM), PRINT_FIELD(DSO),
+ show_symbol_offset);
}

printf("\n");
@@ -1084,6 +1090,8 @@ static const struct option options[] = {
OPT_CALLBACK('f', "fields", NULL, "str",
"comma separated output fields prepend with 'type:'. Valid types: hw,sw,trace,raw. Fields: comm,tid,pid,time,cpu,event,trace,ip,sym,dso,addr",
parse_output_fields),
+ OPT_BOOLEAN('\0', "show-symbol-offset", &show_symbol_offset,
+ "Show the offset of the symbol."),

OPT_END()
};
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 632266f..bb522ed 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1205,10 +1205,11 @@ struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
void perf_session__print_ip(union perf_event *event,
struct perf_sample *sample,
struct perf_session *session,
- int print_sym, int print_dso)
+ int print_sym, int print_dso, int print_offset)
{
struct addr_location al;
const char *symname, *dsoname;
+ unsigned long offset;
struct callchain_cursor *cursor = &session->callchain_cursor;
struct callchain_cursor_node *node;

@@ -1259,12 +1260,15 @@ void perf_session__print_ip(union perf_event *event,
} else {
printf("%16" PRIx64, sample->ip);
if (print_sym) {
- if (al.sym && al.sym->name)
+ printf(" ");
+ if (al.sym && al.sym->name) {
symname = al.sym->name;
- else
- symname = "";
-
- printf(" %s", symname);
+ offset = al.addr - al.sym->start;
+ if (print_offset)
+ printf("%s+0x%lx", symname, offset);
+ else
+ printf("%s", symname);
+ }
}

if (print_dso) {
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index de4178d..8187e68 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -170,6 +170,7 @@ struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
void perf_session__print_ip(union perf_event *event,
struct perf_sample *sample,
struct perf_session *session,
- int print_sym, int print_dso);
+ int print_sym, int print_dso,
+ int print_offset);

#endif /* __PERF_SESSION_H */

2011-06-28 07:40:33

by Akihiro Nagai

[permalink] [raw]
Subject: [PATCH -tip 4/4] perf script: add option resolving vmlinux path

Add the option get the path of [kernel.kallsyms].
Specify '--show-kernel-path' option to use this function.
This patch enables other applications to use this output easily.

Without --show-kernel-path option

# perf script -f ip,dso
ffffffff81467612 irq_return ([kernel.kallsyms])
ffffffff81467612 irq_return ([kernel.kallsyms])
7f24fc02a6b3 _start (/lib64/ld-2.14.so)
[snip]

With --show-kernel-path option

# perf script -f ip,dso --show-kernel-path
ffffffff81467612 irq_return (/lib/modules/3.0.0-rc3-tip+/build/vmlinux)
ffffffff81467612 irq_return (/lib/modules/3.0.0-rc3-tip+/build/vmlinux)
7f24fc02a6b3 _start (/lib64/ld-2.14.so)
[snip]

Signed-off-by: Akihiro Nagai <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Frederic Weisbecker <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: David Ahern <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
---

tools/perf/Documentation/perf-script.txt | 3 +++
tools/perf/builtin-script.c | 15 +++++----------
tools/perf/util/session.c | 26 +++++++++-----------------
tools/perf/util/symbol.c | 12 ++++++++++++
tools/perf/util/symbol.h | 2 ++
5 files changed, 31 insertions(+), 27 deletions(-)

diff --git a/tools/perf/Documentation/perf-script.txt b/tools/perf/Documentation/perf-script.txt
index 39c4721..be59d8b 100644
--- a/tools/perf/Documentation/perf-script.txt
+++ b/tools/perf/Documentation/perf-script.txt
@@ -185,6 +185,9 @@ OPTIONS
--show-symbol-offset::
Show the offset of the symbols.

+--show-kernel-path::
+ Try to resolve the path of [kernel.kallsyms]
+
SEE ALSO
--------
linkperf:perf-record[1], linkperf:perf-script-perl[1],
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index f9b7047..750673f 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -324,7 +324,7 @@ static void print_sample_addr(union perf_event *event,
{
struct addr_location al;
u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
- const char *symname, *dsoname;
+ const char *symname;
unsigned long offset;

printf("%16" PRIx64, sample->addr);
@@ -359,14 +359,8 @@ static void print_sample_addr(union perf_event *event,
}
}

- if (PRINT_FIELD(DSO)) {
- if (al.map && al.map->dso && al.map->dso->name)
- dsoname = al.map->dso->name;
- else
- dsoname = "";
-
- printf(" (%s)", dsoname);
- }
+ if (PRINT_FIELD(DSO) && al.map && al.map->dso)
+ printf(" (%s)", dso__get_dsoname_for_print(al.map->dso));
}

static void process_event(union perf_event *event __unused,
@@ -1092,7 +1086,8 @@ static const struct option options[] = {
parse_output_fields),
OPT_BOOLEAN('\0', "show-symbol-offset", &show_symbol_offset,
"Show the offset of the symbol."),
-
+ OPT_BOOLEAN('\0', "show-kernel-path", &show_kernel_path,
+ "Show the path of [kernel.kallsyms]"),
OPT_END()
};

diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index bb522ed..f385511 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -13,6 +13,8 @@
#include "sort.h"
#include "util.h"

+bool show_kernel_path;
+
static int perf_session__open(struct perf_session *self, bool force)
{
struct stat input_stat;
@@ -1208,7 +1210,7 @@ void perf_session__print_ip(union perf_event *event,
int print_sym, int print_dso, int print_offset)
{
struct addr_location al;
- const char *symname, *dsoname;
+ const char *symname;
unsigned long offset;
struct callchain_cursor *cursor = &session->callchain_cursor;
struct callchain_cursor_node *node;
@@ -1244,14 +1246,9 @@ void perf_session__print_ip(union perf_event *event,

printf(" %s", symname);
}
- if (print_dso) {
- if (node->map && node->map->dso && node->map->dso->name)
- dsoname = node->map->dso->name;
- else
- dsoname = "";
-
- printf(" (%s)", dsoname);
- }
+ if (print_dso && node->map && node->map->dso)
+ printf(" (%s)", dso__get_dsoname_for_print(
+ node->map->dso));
printf("\n");

callchain_cursor_advance(cursor);
@@ -1271,13 +1268,8 @@ void perf_session__print_ip(union perf_event *event,
}
}

- if (print_dso) {
- if (al.map && al.map->dso && al.map->dso->name)
- dsoname = al.map->dso->name;
- else
- dsoname = "";
-
- printf(" (%s)", dsoname);
- }
+ if (print_dso && al.map && al.map->dso)
+ printf(" (%s)",
+ dso__get_dsoname_for_print(al.map->dso));
}
}
diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index eec1963..60dbd39 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -41,6 +41,7 @@ static int dso__load_guest_kernel_sym(struct dso *dso, struct map *map,
symbol_filter_t filter);
static int vmlinux_path__nr_entries;
static char **vmlinux_path;
+bool show_kernel_path;

struct symbol_conf symbol_conf = {
.exclude_other = true,
@@ -431,6 +432,17 @@ size_t dso__fprintf(struct dso *dso, enum map_type type, FILE *fp)
return ret;
}

+const char *dso__get_dsoname_for_print(struct dso *dso)
+{
+ if (dso) {
+ if (show_kernel_path && dso->long_name)
+ return dso->long_name;
+ else
+ return dso->name;
+ } else
+ return "";
+}
+
int kallsyms__parse(const char *filename, void *arg,
int (*process_symbol)(void *arg, const char *name,
char type, u64 start, u64 end))
diff --git a/tools/perf/util/symbol.h b/tools/perf/util/symbol.h
index 325ee36..6b9a7f3 100644
--- a/tools/perf/util/symbol.h
+++ b/tools/perf/util/symbol.h
@@ -96,6 +96,7 @@ struct symbol_conf {
};

extern struct symbol_conf symbol_conf;
+extern bool show_kernel_path;

static inline void *symbol__priv(struct symbol *sym)
{
@@ -195,6 +196,7 @@ size_t dso__fprintf_buildid(struct dso *dso, FILE *fp);
size_t dso__fprintf_symbols_by_name(struct dso *dso,
enum map_type type, FILE *fp);
size_t dso__fprintf(struct dso *dso, enum map_type type, FILE *fp);
+const char *dso__get_dsoname_for_print(struct dso *dso);

enum symtab_type {
SYMTAB__KALLSYMS = 0,

2011-06-28 14:53:21

by David Ahern

[permalink] [raw]
Subject: Re: [PATCH -tip 3/4] perf script: add the option to show the offset of symbols

On 06/28/2011 01:40 AM, Akihiro Nagai wrote:
> Add the option '--show-symbol-offset' to show the offset from
> the symbols in the output of perf-script. We can get the more
> detailed address information.

I like the offset option; added that locally last week as well. more below.

>
> Output sample:
> # perf script -f ip,addr,sym --show-symbol-offset
> 301ec016b0 ffffffff81467612 irq_return+0x0
> 301ec016b0 _start+0x0 ffffffff81467612 irq_return+0x0
> 301ec04b70 _dl_start+0x0 301ec016b3 _start+0x3
> 301ec04b70 _dl_start+0x0 ffffffff81467612 irq_return+0x0
> 301ec04b96 _dl_start+0x26 ffffffff81467612 irq_return+0x0
> 301ec04b9d _dl_start+0x2d ffffffff81467612 irq_return+0x0
> 301ec04c0d _dl_start+0x9d 301ec04beb _dl_start+0x7b
> 301ec04bf0 _dl_start+0x80 301ec04c11 _dl_start+0xa1
> [snip]
>
> Signed-off-by: Akihiro Nagai <[email protected]>
> Cc: Peter Zijlstra <[email protected]>
> Cc: Frederic Weisbecker <[email protected]>
> Cc: Paul Mackerras <[email protected]>
> Cc: Ingo Molnar <[email protected]>
> Cc: Arnaldo Carvalho de Melo <[email protected]>
> Cc: David Ahern <[email protected]>
> Cc: Masami Hiramatsu <[email protected]>
> ---
>
> tools/perf/Documentation/perf-script.txt | 3 +++
> tools/perf/builtin-script.c | 20 ++++++++++++++------
> tools/perf/util/session.c | 16 ++++++++++------
> tools/perf/util/session.h | 3 ++-
> 4 files changed, 29 insertions(+), 13 deletions(-)
>
> diff --git a/tools/perf/Documentation/perf-script.txt b/tools/perf/Documentation/perf-script.txt
> index c6068cb..39c4721 100644
> --- a/tools/perf/Documentation/perf-script.txt
> +++ b/tools/perf/Documentation/perf-script.txt
> @@ -182,6 +182,9 @@ OPTIONS
> --hide-call-graph::
> When printing symbols do not display call chain.
>
> +--show-symbol-offset::
> + Show the offset of the symbols.
> +
> SEE ALSO
> --------
> linkperf:perf-record[1], linkperf:perf-script-perl[1],
> diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
> index 2d68086..f9b7047 100644
> --- a/tools/perf/builtin-script.c
> +++ b/tools/perf/builtin-script.c
> @@ -21,6 +21,7 @@ static u64 last_timestamp;
> static u64 nr_unordered;
> extern const struct option record_options[];
> static bool no_callchain;
> +static bool show_symbol_offset;
>
> enum perf_output_field {
> PERF_OUTPUT_COMM = 1U << 0,
> @@ -324,6 +325,7 @@ static void print_sample_addr(union perf_event *event,
> struct addr_location al;
> u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
> const char *symname, *dsoname;
> + unsigned long offset;
>
> printf("%16" PRIx64, sample->addr);
>
> @@ -346,12 +348,15 @@ static void print_sample_addr(union perf_event *event,
> al.sym = map__find_symbol(al.map, al.addr, NULL);
>
> if (PRINT_FIELD(SYM)) {
> - if (al.sym && al.sym->name)
> + printf(" ");
> + if (al.sym && al.sym->name) {
> symname = al.sym->name;
> - else
> - symname = "";

dropping the else path will cause columns to misalign when ip's do not
resolve.

> -
> - printf(" %16s", symname);
> + offset = al.addr - al.sym->start;

move the offset calculation under the if (show_symbol_offset)

> + if (show_symbol_offset)
> + printf("%16s+0x%lx", symname, offset);
> + else
> + printf("%16s", symname);
> + }
> }
>
> if (PRINT_FIELD(DSO)) {
> @@ -390,7 +395,8 @@ static void process_event(union perf_event *event __unused,
> else
> printf("\n");
> perf_session__print_ip(event, sample, session,
> - PRINT_FIELD(SYM), PRINT_FIELD(DSO));
> + PRINT_FIELD(SYM), PRINT_FIELD(DSO),
> + show_symbol_offset);
> }
>
> printf("\n");
> @@ -1084,6 +1090,8 @@ static const struct option options[] = {
> OPT_CALLBACK('f', "fields", NULL, "str",
> "comma separated output fields prepend with 'type:'. Valid types: hw,sw,trace,raw. Fields: comm,tid,pid,time,cpu,event,trace,ip,sym,dso,addr",
> parse_output_fields),
> + OPT_BOOLEAN('\0', "show-symbol-offset", &show_symbol_offset,
> + "Show the offset of the symbol."),
>
> OPT_END()
> };
> diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
> index 632266f..bb522ed 100644
> --- a/tools/perf/util/session.c
> +++ b/tools/perf/util/session.c
> @@ -1205,10 +1205,11 @@ struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
> void perf_session__print_ip(union perf_event *event,
> struct perf_sample *sample,
> struct perf_session *session,
> - int print_sym, int print_dso)
> + int print_sym, int print_dso, int print_offset)
> {
> struct addr_location al;
> const char *symname, *dsoname;
> + unsigned long offset;
> struct callchain_cursor *cursor = &session->callchain_cursor;
> struct callchain_cursor_node *node;
>
> @@ -1259,12 +1260,15 @@ void perf_session__print_ip(union perf_event *event,
> } else {
> printf("%16" PRIx64, sample->ip);
> if (print_sym) {
> - if (al.sym && al.sym->name)
> + printf(" ");
> + if (al.sym && al.sym->name) {
> symname = al.sym->name;
> - else
> - symname = "";
> -
> - printf(" %s", symname);
> + offset = al.addr - al.sym->start;
> + if (print_offset)
> + printf("%s+0x%lx", symname, offset);

same comments for this code path.

David


> + else
> + printf("%s", symname);
> + }
> }
>
> if (print_dso) {
> diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
> index de4178d..8187e68 100644
> --- a/tools/perf/util/session.h
> +++ b/tools/perf/util/session.h
> @@ -170,6 +170,7 @@ struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
> void perf_session__print_ip(union perf_event *event,
> struct perf_sample *sample,
> struct perf_session *session,
> - int print_sym, int print_dso);
> + int print_sym, int print_dso,
> + int print_offset);
>
> #endif /* __PERF_SESSION_H */
>

2011-06-28 15:00:17

by David Ahern

[permalink] [raw]
Subject: Re: [PATCH -tip 3/4] perf script: add the option to show the offset of symbols

On 06/28/2011 08:52 AM, David Ahern wrote:
> On 06/28/2011 01:40 AM, Akihiro Nagai wrote:
>> Add the option '--show-symbol-offset' to show the offset from
>> the symbols in the output of perf-script. We can get the more
>> detailed address information.
>
> I like the offset option; added that locally last week as well. more below.

one more comment: move the --show-symbol-offset option to a field
parameter - like ip, addr, dso and sym. For parameter checking make it
depend on the sym field.

Update the Documentation file as well.

David

>
>>
>> Output sample:
>> # perf script -f ip,addr,sym --show-symbol-offset
>> 301ec016b0 ffffffff81467612 irq_return+0x0
>> 301ec016b0 _start+0x0 ffffffff81467612 irq_return+0x0
>> 301ec04b70 _dl_start+0x0 301ec016b3 _start+0x3
>> 301ec04b70 _dl_start+0x0 ffffffff81467612 irq_return+0x0
>> 301ec04b96 _dl_start+0x26 ffffffff81467612 irq_return+0x0
>> 301ec04b9d _dl_start+0x2d ffffffff81467612 irq_return+0x0
>> 301ec04c0d _dl_start+0x9d 301ec04beb _dl_start+0x7b
>> 301ec04bf0 _dl_start+0x80 301ec04c11 _dl_start+0xa1
>> [snip]
>>
>> Signed-off-by: Akihiro Nagai <[email protected]>
>> Cc: Peter Zijlstra <[email protected]>
>> Cc: Frederic Weisbecker <[email protected]>
>> Cc: Paul Mackerras <[email protected]>
>> Cc: Ingo Molnar <[email protected]>
>> Cc: Arnaldo Carvalho de Melo <[email protected]>
>> Cc: David Ahern <[email protected]>
>> Cc: Masami Hiramatsu <[email protected]>
>> ---
>>
>> tools/perf/Documentation/perf-script.txt | 3 +++
>> tools/perf/builtin-script.c | 20 ++++++++++++++------
>> tools/perf/util/session.c | 16 ++++++++++------
>> tools/perf/util/session.h | 3 ++-
>> 4 files changed, 29 insertions(+), 13 deletions(-)
>>
>> diff --git a/tools/perf/Documentation/perf-script.txt b/tools/perf/Documentation/perf-script.txt
>> index c6068cb..39c4721 100644
>> --- a/tools/perf/Documentation/perf-script.txt
>> +++ b/tools/perf/Documentation/perf-script.txt
>> @@ -182,6 +182,9 @@ OPTIONS
>> --hide-call-graph::
>> When printing symbols do not display call chain.
>>
>> +--show-symbol-offset::
>> + Show the offset of the symbols.
>> +
>> SEE ALSO
>> --------
>> linkperf:perf-record[1], linkperf:perf-script-perl[1],
>> diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
>> index 2d68086..f9b7047 100644
>> --- a/tools/perf/builtin-script.c
>> +++ b/tools/perf/builtin-script.c
>> @@ -21,6 +21,7 @@ static u64 last_timestamp;
>> static u64 nr_unordered;
>> extern const struct option record_options[];
>> static bool no_callchain;
>> +static bool show_symbol_offset;
>>
>> enum perf_output_field {
>> PERF_OUTPUT_COMM = 1U << 0,
>> @@ -324,6 +325,7 @@ static void print_sample_addr(union perf_event *event,
>> struct addr_location al;
>> u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
>> const char *symname, *dsoname;
>> + unsigned long offset;
>>
>> printf("%16" PRIx64, sample->addr);
>>
>> @@ -346,12 +348,15 @@ static void print_sample_addr(union perf_event *event,
>> al.sym = map__find_symbol(al.map, al.addr, NULL);
>>
>> if (PRINT_FIELD(SYM)) {
>> - if (al.sym && al.sym->name)
>> + printf(" ");
>> + if (al.sym && al.sym->name) {
>> symname = al.sym->name;
>> - else
>> - symname = "";
>
> dropping the else path will cause columns to misalign when ip's do not
> resolve.
>
>> -
>> - printf(" %16s", symname);
>> + offset = al.addr - al.sym->start;
>
> move the offset calculation under the if (show_symbol_offset)
>
>> + if (show_symbol_offset)
>> + printf("%16s+0x%lx", symname, offset);
>> + else
>> + printf("%16s", symname);
>> + }
>> }
>>
>> if (PRINT_FIELD(DSO)) {
>> @@ -390,7 +395,8 @@ static void process_event(union perf_event *event __unused,
>> else
>> printf("\n");
>> perf_session__print_ip(event, sample, session,
>> - PRINT_FIELD(SYM), PRINT_FIELD(DSO));
>> + PRINT_FIELD(SYM), PRINT_FIELD(DSO),
>> + show_symbol_offset);
>> }
>>
>> printf("\n");
>> @@ -1084,6 +1090,8 @@ static const struct option options[] = {
>> OPT_CALLBACK('f', "fields", NULL, "str",
>> "comma separated output fields prepend with 'type:'. Valid types: hw,sw,trace,raw. Fields: comm,tid,pid,time,cpu,event,trace,ip,sym,dso,addr",
>> parse_output_fields),
>> + OPT_BOOLEAN('\0', "show-symbol-offset", &show_symbol_offset,
>> + "Show the offset of the symbol."),
>>
>> OPT_END()
>> };
>> diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
>> index 632266f..bb522ed 100644
>> --- a/tools/perf/util/session.c
>> +++ b/tools/perf/util/session.c
>> @@ -1205,10 +1205,11 @@ struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
>> void perf_session__print_ip(union perf_event *event,
>> struct perf_sample *sample,
>> struct perf_session *session,
>> - int print_sym, int print_dso)
>> + int print_sym, int print_dso, int print_offset)
>> {
>> struct addr_location al;
>> const char *symname, *dsoname;
>> + unsigned long offset;
>> struct callchain_cursor *cursor = &session->callchain_cursor;
>> struct callchain_cursor_node *node;
>>
>> @@ -1259,12 +1260,15 @@ void perf_session__print_ip(union perf_event *event,
>> } else {
>> printf("%16" PRIx64, sample->ip);
>> if (print_sym) {
>> - if (al.sym && al.sym->name)
>> + printf(" ");
>> + if (al.sym && al.sym->name) {
>> symname = al.sym->name;
>> - else
>> - symname = "";
>> -
>> - printf(" %s", symname);
>> + offset = al.addr - al.sym->start;
>> + if (print_offset)
>> + printf("%s+0x%lx", symname, offset);
>
> same comments for this code path.
>
> David
>
>
>> + else
>> + printf("%s", symname);
>> + }
>> }
>>
>> if (print_dso) {
>> diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
>> index de4178d..8187e68 100644
>> --- a/tools/perf/util/session.h
>> +++ b/tools/perf/util/session.h
>> @@ -170,6 +170,7 @@ struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
>> void perf_session__print_ip(union perf_event *event,
>> struct perf_sample *sample,
>> struct perf_session *session,
>> - int print_sym, int print_dso);
>> + int print_sym, int print_dso,
>> + int print_offset);
>>
>> #endif /* __PERF_SESSION_H */
>>

2011-06-28 15:22:22

by David Ahern

[permalink] [raw]
Subject: Re: [PATCH -tip 0/4] perf script: add BTS analysis features

On 06/28/2011 01:40 AM, Akihiro Nagai wrote:
> BTW, I couldn't run a script generated by 'perf script -g {pyton|perl}'
> with branch events. How do I make scripts analyzing branch events?
>
> # perf record -e branches:u -c 1 -d ls
> # perf script -g python
> # perf script -s perf-script.py
> in trace_begin
> Fatal: no event_list!
> # perf script -g perl
> # perf script -s perf-script.pl
> Fatal: no event_list!

Generated scripts currently only work on trace events. At this point it
is arguable that perf-script should be focused on the python and perl
scripts and the code for dumping the events should be moved to a new
command (perf-dump?).

David

2011-06-30 04:44:46

by Akihiro Nagai

[permalink] [raw]
Subject: Re: [PATCH -tip 3/4] perf script: add the option to show the offset of symbols

(2011/06/28 23:55), David Ahern wrote:
> On 06/28/2011 08:52 AM, David Ahern wrote:
>> On 06/28/2011 01:40 AM, Akihiro Nagai wrote:
>>> Add the option '--show-symbol-offset' to show the offset from
>>> the symbols in the output of perf-script. We can get the more
>>> detailed address information.
>>
>> I like the offset option; added that locally last week as well. more below.
>
> one more comment: move the --show-symbol-offset option to a field
> parameter - like ip, addr, dso and sym. For parameter checking make it
> depend on the sym field.
>
> Update the Documentation file as well.
OK, I'm going to add 'offs' field specifier instead of
the option '--show-symbol-offset', and update documents.

>
> David
>
>>
>>>
>>> Output sample:
>>> # perf script -f ip,addr,sym --show-symbol-offset
>>> 301ec016b0 ffffffff81467612 irq_return+0x0
>>> 301ec016b0 _start+0x0 ffffffff81467612 irq_return+0x0
>>> 301ec04b70 _dl_start+0x0 301ec016b3 _start+0x3
>>> 301ec04b70 _dl_start+0x0 ffffffff81467612 irq_return+0x0
>>> 301ec04b96 _dl_start+0x26 ffffffff81467612 irq_return+0x0
>>> 301ec04b9d _dl_start+0x2d ffffffff81467612 irq_return+0x0
>>> 301ec04c0d _dl_start+0x9d 301ec04beb _dl_start+0x7b
>>> 301ec04bf0 _dl_start+0x80 301ec04c11 _dl_start+0xa1
>>> [snip]
>>>
>>> Signed-off-by: Akihiro Nagai<[email protected]>
>>> Cc: Peter Zijlstra<[email protected]>
>>> Cc: Frederic Weisbecker<[email protected]>
>>> Cc: Paul Mackerras<[email protected]>
>>> Cc: Ingo Molnar<[email protected]>
>>> Cc: Arnaldo Carvalho de Melo<[email protected]>
>>> Cc: David Ahern<[email protected]>
>>> Cc: Masami Hiramatsu<[email protected]>
>>> ---
>>>
>>> tools/perf/Documentation/perf-script.txt | 3 +++
>>> tools/perf/builtin-script.c | 20 ++++++++++++++------
>>> tools/perf/util/session.c | 16 ++++++++++------
>>> tools/perf/util/session.h | 3 ++-
>>> 4 files changed, 29 insertions(+), 13 deletions(-)
>>>
>>> diff --git a/tools/perf/Documentation/perf-script.txt b/tools/perf/Documentation/perf-script.txt
>>> index c6068cb..39c4721 100644
>>> --- a/tools/perf/Documentation/perf-script.txt
>>> +++ b/tools/perf/Documentation/perf-script.txt
>>> @@ -182,6 +182,9 @@ OPTIONS
>>> --hide-call-graph::
>>> When printing symbols do not display call chain.
>>>
>>> +--show-symbol-offset::
>>> + Show the offset of the symbols.
>>> +
>>> SEE ALSO
>>> --------
>>> linkperf:perf-record[1], linkperf:perf-script-perl[1],
>>> diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
>>> index 2d68086..f9b7047 100644
>>> --- a/tools/perf/builtin-script.c
>>> +++ b/tools/perf/builtin-script.c
>>> @@ -21,6 +21,7 @@ static u64 last_timestamp;
>>> static u64 nr_unordered;
>>> extern const struct option record_options[];
>>> static bool no_callchain;
>>> +static bool show_symbol_offset;
>>>
>>> enum perf_output_field {
>>> PERF_OUTPUT_COMM = 1U<< 0,
>>> @@ -324,6 +325,7 @@ static void print_sample_addr(union perf_event *event,
>>> struct addr_location al;
>>> u8 cpumode = event->header.misc& PERF_RECORD_MISC_CPUMODE_MASK;
>>> const char *symname, *dsoname;
>>> + unsigned long offset;
>>>
>>> printf("%16" PRIx64, sample->addr);
>>>
>>> @@ -346,12 +348,15 @@ static void print_sample_addr(union perf_event *event,
>>> al.sym = map__find_symbol(al.map, al.addr, NULL);
>>>
>>> if (PRINT_FIELD(SYM)) {
>>> - if (al.sym&& al.sym->name)
>>> + printf(" ");
>>> + if (al.sym&& al.sym->name) {
>>> symname = al.sym->name;
>>> - else
>>> - symname = "";
>>
>> dropping the else path will cause columns to misalign when ip's do not
>> resolve.
I see.
I'd like to add a magic word that indicates perf-script failed to resolve the symbol.
For example, '[unknown]'.

>>
>>> -
>>> - printf(" %16s", symname);
>>> + offset = al.addr - al.sym->start;
>>
>> move the offset calculation under the if (show_symbol_offset)
OK.

>>
>>> + if (show_symbol_offset)
>>> + printf("%16s+0x%lx", symname, offset);
>>> + else
>>> + printf("%16s", symname);
>>> + }
>>> }
>>>
>>> if (PRINT_FIELD(DSO)) {
>>> @@ -390,7 +395,8 @@ static void process_event(union perf_event *event __unused,
>>> else
>>> printf("\n");
>>> perf_session__print_ip(event, sample, session,
>>> - PRINT_FIELD(SYM), PRINT_FIELD(DSO));
>>> + PRINT_FIELD(SYM), PRINT_FIELD(DSO),
>>> + show_symbol_offset);
>>> }
>>>
>>> printf("\n");
>>> @@ -1084,6 +1090,8 @@ static const struct option options[] = {
>>> OPT_CALLBACK('f', "fields", NULL, "str",
>>> "comma separated output fields prepend with 'type:'. Valid types: hw,sw,trace,raw. Fields: comm,tid,pid,time,cpu,event,trace,ip,sym,dso,addr",
>>> parse_output_fields),
>>> + OPT_BOOLEAN('\0', "show-symbol-offset",&show_symbol_offset,
>>> + "Show the offset of the symbol."),
>>>
>>> OPT_END()
>>> };
>>> diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
>>> index 632266f..bb522ed 100644
>>> --- a/tools/perf/util/session.c
>>> +++ b/tools/perf/util/session.c
>>> @@ -1205,10 +1205,11 @@ struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
>>> void perf_session__print_ip(union perf_event *event,
>>> struct perf_sample *sample,
>>> struct perf_session *session,
>>> - int print_sym, int print_dso)
>>> + int print_sym, int print_dso, int print_offset)
>>> {
>>> struct addr_location al;
>>> const char *symname, *dsoname;
>>> + unsigned long offset;
>>> struct callchain_cursor *cursor =&session->callchain_cursor;
>>> struct callchain_cursor_node *node;
>>>
>>> @@ -1259,12 +1260,15 @@ void perf_session__print_ip(union perf_event *event,
>>> } else {
>>> printf("%16" PRIx64, sample->ip);
>>> if (print_sym) {
>>> - if (al.sym&& al.sym->name)
>>> + printf(" ");
>>> + if (al.sym&& al.sym->name) {
>>> symname = al.sym->name;
>>> - else
>>> - symname = "";
>>> -
>>> - printf(" %s", symname);
>>> + offset = al.addr - al.sym->start;
>>> + if (print_offset)
>>> + printf("%s+0x%lx", symname, offset);
>>
>> same comments for this code path.
OK. I'm going to move the code.


Thank you.

>>
>> David
>>
>>
>>> + else
>>> + printf("%s", symname);
>>> + }
>>> }
>>>
>>> if (print_dso) {
>>> diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
>>> index de4178d..8187e68 100644
>>> --- a/tools/perf/util/session.h
>>> +++ b/tools/perf/util/session.h
>>> @@ -170,6 +170,7 @@ struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
>>> void perf_session__print_ip(union perf_event *event,
>>> struct perf_sample *sample,
>>> struct perf_session *session,
>>> - int print_sym, int print_dso);
>>> + int print_sym, int print_dso,
>>> + int print_offset);
>>>
>>> #endif /* __PERF_SESSION_H */
>>>

2011-06-30 05:19:10

by Akihiro Nagai

[permalink] [raw]
Subject: Re: [PATCH -tip 0/4] perf script: add BTS analysis features

(2011/06/29 0:21), David Ahern wrote:
> On 06/28/2011 01:40 AM, Akihiro Nagai wrote:
>> BTW, I couldn't run a script generated by 'perf script -g {pyton|perl}'
>> with branch events. How do I make scripts analyzing branch events?
>>
>> # perf record -e branches:u -c 1 -d ls
>> # perf script -g python
>> # perf script -s perf-script.py
>> in trace_begin
>> Fatal: no event_list!
>> # perf script -g perl
>> # perf script -s perf-script.pl
>> Fatal: no event_list!
>
> Generated scripts currently only work on trace events. At this point it
> is arguable that perf-script should be focused on the python and perl
> scripts and the code for dumping the events should be moved to a new
> command (perf-dump?).
I see. Thank you for your information.
I'd like to try adding the function to deal with HW-events with perf-script.

I like perf-dump idea. It makes easier to specialize output format
based on each event type.

Thank you.
>
> David

2011-06-30 15:42:49

by David Ahern

[permalink] [raw]
Subject: Re: [PATCH -tip 3/4] perf script: add the option to show the offset of symbols

On 06/29/2011 10:44 PM, Akihiro Nagai wrote:
>>>> @@ -346,12 +348,15 @@ static void print_sample_addr(union perf_event
>>>> *event,
>>>> al.sym = map__find_symbol(al.map, al.addr, NULL);
>>>>
>>>> if (PRINT_FIELD(SYM)) {
>>>> - if (al.sym&& al.sym->name)
>>>> + printf(" ");
>>>> + if (al.sym&& al.sym->name) {
>>>> symname = al.sym->name;
>>>> - else
>>>> - symname = "";
>>>
>>> dropping the else path will cause columns to misalign when ip's do not
>>> resolve.
> I see.
> I'd like to add a magic word that indicates perf-script failed to
> resolve the symbol.
> For example, '[unknown]'.

ok. constant number of fields would make secondary analysis tools easier.

David