Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [RFC] perf script: add flamegraph.py script
From:   Andreas Gerstmayr <agerstmayr@redhat.com>
To:     Jiri Olsa <jolsa@redhat.com>
Cc:     linux-kernel@vger.kernel.org, kabbott@redhat.com,
        skozina@redhat.com, mpetlan@redhat.com, nathans@redhat.com,
        mgoodwin@redhat.com, linux-perf-users@vger.kernel.org,
        bgregg@netflix.com, mspier@netflix.com,
        Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Arnaldo Carvalho de Melo <acme@kernel.org>,
        Mark Rutland <mark.rutland@arm.com>,
        Alexander Shishkin <alexander.shishkin@linux.intel.com>,
        Namhyung Kim <namhyung@kernel.org>
References: <20200221175500.83774-1-agerstmayr@redhat.com>
 <20200225195418.GA160300@krava>
 <0582d729-0e07-b95d-7cad-8912514b8871@redhat.com>
 <20200225202056.GA160294@krava>
 <cd44bf7a-6766-d42b-50d1-482d389fe488@redhat.com>
Message-ID: <47ffe459-0920-60d0-b6b0-d7cebb77448b@redhat.com>
Date:   Thu, 27 Feb 2020 21:16:44 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.5.0
MIME-Version: 1.0
In-Reply-To: <cd44bf7a-6766-d42b-50d1-482d389fe488@redhat.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 25.02.20 21:36, Andreas Gerstmayr wrote:
> On 25.02.20 21:20, Jiri Olsa wrote:
>> On Tue, Feb 25, 2020 at 09:03:19PM +0100, Andreas Gerstmayr wrote:
>>> On 25.02.20 20:54, Jiri Olsa wrote:
>>>> On Fri, Feb 21, 2020 at 06:55:01PM +0100, Andreas Gerstmayr wrote:
>>>>> Usage:
>>>>>
>>>>>       perf script flamegraph -a -F 99 sleep 60
>>>>>
>>>>> Alternative usage:
>>>>>
>>>>>       perf record -a -g -F 99 sleep 60
>>>>>       perf script report flamegraph
>>>>
>>>> nice, could this output the output file, like:
>>>>
>>>>        # perf script report flamegraph --output krava.html
>>>>        dumping data to krava.html
>>
>> I meant the actual line ^^^^, saying that it's writing to the file
> 
> Ah! Sorry, I misunderstood.
> Yep, sure, I can add that.
> 
> I also have one other change lined up to reduce the JSON output, and I'm 
> testing it with huge flamegraphs right now. Will send an update this week.

Ok, I'll need some more time to properly test this with huge flame 
graphs and different browsers. I'll get back to this in ~2 weeks after 
my PTO.


Cheers,
Andreas


> 
> 
> Cheers,
> Andreas
> 
> 
>>
>> thanks,
>> jirka
>>
>>>>
>>>> or something in that sense
>>>>
>>>> other than that it looks good to me
>>>
>>> Yes, it's already implemented.
>>>
>>> $ perf script report flamegraph --output krava.html
>>>
>>> writes the output to krava.html
>>>
>>> $ perf script report flamegraph --help
>>>
>>> shows the supported arguments.
>>>
>>> The only gotcha is that you need to have a perf.data in the same 
>>> directory
>>> when calling this command, otherwise perf complains about a missing
>>> perf.data and doesn't call the flamegraph.py script.
>>>
>>>
>>> Cheers,
>>> Andreas
>>>
>>>
>>>>
>>>> thanks,
>>>> jirka
>>>>
>>>>
>>>>>
>>>>> Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
>>>>> Cc: Peter Zijlstra <peterz@infradead.org>
>>>>> Cc: Ingo Molnar <mingo@redhat.com>
>>>>> Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
>>>>> Cc: Mark Rutland <mark.rutland@arm.com>
>>>>> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
>>>>> Cc: Jiri Olsa <jolsa@redhat.com>
>>>>> Cc: Namhyung Kim <namhyung@kernel.org>
>>>>> ---
>>>>>
>>>>> I'm currently preparing packages for d3-flame-graph. For Fedora, 
>>>>> the copr
>>>>> at
>>>>> https://copr.fedorainfracloud.org/coprs/agerstmayr/reviews/package/js-d3-flame-graph/ 
>>>>>
>>>>> can be installed, or alternatively the prebuilt standalone 
>>>>> d3-flame-graph
>>>>> template can be downloaded from
>>>>> https://raw.githubusercontent.com/andreasgerstmayr/specs/master/reviews/js-d3-flame-graph/template.html 
>>>>>
>>>>> and moved into /usr/share/d3-flame-graph/template.html
>>>>>
>>>>>    .../perf/scripts/python/bin/flamegraph-record |   2 +
>>>>>    .../perf/scripts/python/bin/flamegraph-report |   3 +
>>>>>    tools/perf/scripts/python/flamegraph.py       | 117 
>>>>> ++++++++++++++++++
>>>>>    3 files changed, 122 insertions(+)
>>>>>    create mode 100755 tools/perf/scripts/python/bin/flamegraph-record
>>>>>    create mode 100755 tools/perf/scripts/python/bin/flamegraph-report
>>>>>    create mode 100755 tools/perf/scripts/python/flamegraph.py
>>>>>
>>>>> diff --git a/tools/perf/scripts/python/bin/flamegraph-record 
>>>>> b/tools/perf/scripts/python/bin/flamegraph-record
>>>>> new file mode 100755
>>>>> index 000000000000..725d66e71570
>>>>> --- /dev/null
>>>>> +++ b/tools/perf/scripts/python/bin/flamegraph-record
>>>>> @@ -0,0 +1,2 @@
>>>>> +#!/usr/bin/sh
>>>>> +perf record -g "$@"
>>>>> diff --git a/tools/perf/scripts/python/bin/flamegraph-report 
>>>>> b/tools/perf/scripts/python/bin/flamegraph-report
>>>>> new file mode 100755
>>>>> index 000000000000..b1a79afd903b
>>>>> --- /dev/null
>>>>> +++ b/tools/perf/scripts/python/bin/flamegraph-report
>>>>> @@ -0,0 +1,3 @@
>>>>> +#!/usr/bin/sh
>>>>> +# description: create flame graphs
>>>>> +perf script -s "$PERF_EXEC_PATH"/scripts/python/flamegraph.py -- "$@"
>>>>> diff --git a/tools/perf/scripts/python/flamegraph.py 
>>>>> b/tools/perf/scripts/python/flamegraph.py
>>>>> new file mode 100755
>>>>> index 000000000000..2e9139ef2c4a
>>>>> --- /dev/null
>>>>> +++ b/tools/perf/scripts/python/flamegraph.py
>>>>> @@ -0,0 +1,117 @@
>>>>> +# flamegraph.py - create flame graphs from perf samples
>>>>> +# SPDX-License-Identifier: GPL-2.0
>>>>> +#
>>>>> +# Usage:
>>>>> +#
>>>>> +#  perf record -a -g -F 99 sleep 60
>>>>> +#  perf script report flamegraph
>>>>> +#
>>>>> +# Combined data collection and flamegraph generation:
>>>>> +#
>>>>> +#  perf script flamegraph -a -F 99 sleep 60
>>>>> +#
>>>>> +# Written by Andreas Gerstmayr <agerstmayr@redhat.com>
>>>>> +# Flame Graphs invented by Brendan Gregg <bgregg@netflix.com>
>>>>> +# Works in tandem with d3-flame-graph by Martin Spier 
>>>>> <mspier@netflix.com>
>>>>> +
>>>>> +import sys
>>>>> +import os
>>>>> +import argparse
>>>>> +import json
>>>>> +
>>>>> +
>>>>> +class Node:
>>>>> +    def __init__(self, name, libtype=""):
>>>>> +        self.name = name
>>>>> +        self.libtype = libtype
>>>>> +        self.value = 0
>>>>> +        self.children = []
>>>>> +
>>>>> +
>>>>> +class FlameGraphCLI:
>>>>> +    def __init__(self, args):
>>>>> +        self.args = args
>>>>> +        self.stack = Node("root")
>>>>> +
>>>>> +        if self.args.format == "html" and \
>>>>> +                not os.path.isfile(self.args.template):
>>>>> +            print(f"Flame Graph template '{self.args.template}' 
>>>>> does not " +
>>>>> +                  f"exist. Please install the d3-flame-graph 
>>>>> package, " +
>>>>> +                  f"specify an existing flame graph template " +
>>>>> +                  f"(--template PATH) or another output format " +
>>>>> +                  f"(--format FORMAT).", file=sys.stderr)
>>>>> +            sys.exit(1)
>>>>> +
>>>>> +    def find_or_create_node(self, node, name, dso):
>>>>> +        libtype = "kernel" if dso == "[kernel.kallsyms]" else ""
>>>>> +        if name is None:
>>>>> +            name = "[unknown]"
>>>>> +
>>>>> +        for child in node.children:
>>>>> +            if child.name == name and child.libtype == libtype:
>>>>> +                return child
>>>>> +
>>>>> +        child = Node(name, libtype)
>>>>> +        node.children.append(child)
>>>>> +        return child
>>>>> +
>>>>> +    def process_event(self, event):
>>>>> +        node = self.find_or_create_node(self.stack, event["comm"], 
>>>>> None)
>>>>> +        if "callchain" in event:
>>>>> +            for entry in reversed(event['callchain']):
>>>>> +                node = self.find_or_create_node(
>>>>> +                    node, entry.get("sym", {}).get("name"), 
>>>>> event.get("dso"))
>>>>> +        else:
>>>>> +            node = self.find_or_create_node(
>>>>> +                node, entry.get("symbol"), event.get("dso"))
>>>>> +        node.value += 1
>>>>> +
>>>>> +    def trace_end(self):
>>>>> +        def encoder(x): return x.__dict__
>>>>> +        json_str = json.dumps(self.stack, default=encoder,
>>>>> +                              indent=self.args.indent)
>>>>> +
>>>>> +        if self.args.format == "html":
>>>>> +            try:
>>>>> +                with open(self.args.template) as f:
>>>>> +                    output_str = f.read().replace("/** 
>>>>> @flamegraph_params **/",
>>>>> +                                                  json_str)
>>>>> +            except IOError as e:
>>>>> +                print(f"Error reading template file: {e}", 
>>>>> file=sys.stderr)
>>>>> +                sys.exit(1)
>>>>> +            output_fn = self.args.output or "flamegraph.html"
>>>>> +        else:
>>>>> +            output_str = json_str
>>>>> +            output_fn = self.args.output or "stacks.json"
>>>>> +
>>>>> +        if output_fn == "-":
>>>>> +            sys.stdout.write(output_str)
>>>>> +        else:
>>>>> +            try:
>>>>> +                with open(output_fn, "w") as out:
>>>>> +                    out.write(output_str)
>>>>> +            except IOError as e:
>>>>> +                print(f"Error writing output file: {e}", 
>>>>> file=sys.stderr)
>>>>> +                sys.exit(1)
>>>>> +
>>>>> +
>>>>> +if __name__ == "__main__":
>>>>> +    parser = argparse.ArgumentParser(description="Create flame 
>>>>> graphs.")
>>>>> +    parser.add_argument("-F", "--format",
>>>>> +                        default="html", choices=["json", "html"],
>>>>> +                        help="output file format")
>>>>> +    parser.add_argument("-o", "--output",
>>>>> +                        help="output file name")
>>>>> +    parser.add_argument("--indent",
>>>>> +                        type=int, help="JSON indentation")
>>>>> +    parser.add_argument("--template",
>>>>> +                        
>>>>> default="/usr/share/d3-flame-graph/template.html",
>>>>> +                        help="path to flamegraph HTML template")
>>>>> +    parser.add_argument("-i", "--input",
>>>>> +                        help=argparse.SUPPRESS)
>>>>> +
>>>>> +    args = parser.parse_args()
>>>>> +    cli = FlameGraphCLI(args)
>>>>> +
>>>>> +    process_event = cli.process_event
>>>>> +    trace_end = cli.trace_end
>>>>> -- 
>>>>> 2.24.1
>>>>>
>>>>
>>>
>>