Received: by 2002:a25:683:0:0:0:0:0 with SMTP id 125csp1289255ybg; Tue, 2 Jun 2020 06:19:30 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxD5U7EUTHslgWa7d2PaqQInwMQxS0aJB1+vgllG8L25ygyI5XbOX+l8X4Q2obwGltIawj2 X-Received: by 2002:a17:906:17c5:: with SMTP id u5mr22808302eje.275.1591103969820; Tue, 02 Jun 2020 06:19:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1591103969; cv=none; d=google.com; s=arc-20160816; b=aUk4IrX2onAdgtfLjPbm9lu5BS+uNtCkhmvjlPHxtqd5FDy7joiljH8y97BxyXORll VryaJjIweYpCAll1B+2H2UXTGeBpHG+i/3YGgm30YHZxI/hdi2/FpMAgytHqTdKXhl11 TKKldkwFjE5p/+np+YU4JxbaqRupzjBiGEOUG/zd7a/CWl+H0GOCpjUHtOCkgLqF1xsP fD7KuMtD/+HJeWtyjbUe5UYO8JN4DVLlpyN60krJRHa+SMnnVy05/w4Ir5lPzWKTHqXV ZBmPOL1X6qYJAULGpxKFJddNuGPuzciqg0jP/Et5Q1kCofg5ScrmgfVgyhEajMH3XP/n mSPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:ironport-sdr:dkim-signature; bh=1nHLf6GF2DasMM8LcTB/6YcLhSGCeBDmjjbh8PNEsgo=; b=KBnAfIsA96NlJaOoOeSq22mfPgzqAdMr1lz8aoBnRSR8LUwVsSfE7w1D8RWMaVJ0JQ qjoKqAVnXqVvcPnxxKWTSYS4+WjYV70zcBq45JWgVDzg+EPtAJ/MNEcAKaT5A40de0/P T1LtD1D79nOYHCl0Qv/PJ42h08/n29XWOeLB23ZTNC4Djnws0OKSlPpkuM++CLeD8pgM aX/ywR/+sX1Z1w1MDdQN/g1n/C9LvRQ5UOGGoE6btauADHNYDl3nDHe7aqOKcvL/ihmt BAqKDgts0uHsPczzCnkW+E79EZiqge06bWMON1Wlbw31cWasj49kISAxc/dPqljme7WD G1xg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@amazon.com header.s=amazon201209 header.b=HpH48Cxm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id bz17si1392812ejc.605.2020.06.02.06.19.05; Tue, 02 Jun 2020 06:19:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@amazon.com header.s=amazon201209 header.b=HpH48Cxm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727977AbgFBNOk (ORCPT + 99 others); Tue, 2 Jun 2020 09:14:40 -0400 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:44839 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725958AbgFBNOg (ORCPT ); Tue, 2 Jun 2020 09:14:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1591103670; x=1622639670; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=1nHLf6GF2DasMM8LcTB/6YcLhSGCeBDmjjbh8PNEsgo=; b=HpH48Cxmt4zNQTGP59imH0rfxgCLvMejS8wf3z9Gi4FdqcCTUlBY593o zg44M8eRYnnY6KYd6bCckk25ehIpA8O3yCyLewVEvE/Ap+T02MZQqK97p DQSYFo+6jKIf5xNaAtwAPZ/1Ay4wukhZvHxTOYhXf5mpSEKCX9kAKWF5c k=; IronPort-SDR: q4e9Ik0O+VuAPW60fuuyUNwVDOJaLo7ks2vU6mooZSz/VLbIzVWCvhFZRJhyKwIb1Za4ARZY7Y rsOaL59sRAUA== X-IronPort-AV: E=Sophos;i="5.73,464,1583193600"; d="scan'208";a="33879311" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2b-8cc5d68b.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 02 Jun 2020 13:14:26 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2b-8cc5d68b.us-west-2.amazon.com (Postfix) with ESMTPS id 33D15A2461; Tue, 2 Jun 2020 13:14:24 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 2 Jun 2020 13:14:23 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.162.208) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 2 Jun 2020 13:14:06 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH v14 10/15] tools: Add a minimal user-space tool for DAMON Date: Tue, 2 Jun 2020 15:13:48 +0200 Message-ID: <20200602131348.22937-1-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200602130125.20467-1-sjpark@amazon.com> References: <20200602130125.20467-1-sjpark@amazon.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [10.43.162.208] X-ClientProxiedBy: EX13D41UWB002.ant.amazon.com (10.43.161.109) To EX13D31EUA001.ant.amazon.com (10.43.165.15) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: SeongJae Park This commit adds a shallow wrapper python script, ``/tools/damon/damo`` that provides more convenient interface. Note that it is only aimed to be used for minimal reference of the DAMON's debugfs interfaces and for debugging of the DAMON itself. Signed-off-by: SeongJae Park --- tools/damon/.gitignore | 1 + tools/damon/_dist.py | 36 ++++ tools/damon/_recfile.py | 23 +++ tools/damon/bin2txt.py | 67 +++++++ tools/damon/damo | 37 ++++ tools/damon/heats.py | 362 ++++++++++++++++++++++++++++++++++++++ tools/damon/nr_regions.py | 91 ++++++++++ tools/damon/record.py | 212 ++++++++++++++++++++++ tools/damon/report.py | 45 +++++ tools/damon/wss.py | 97 ++++++++++ 10 files changed, 971 insertions(+) create mode 100644 tools/damon/.gitignore create mode 100644 tools/damon/_dist.py create mode 100644 tools/damon/_recfile.py create mode 100644 tools/damon/bin2txt.py create mode 100755 tools/damon/damo create mode 100644 tools/damon/heats.py create mode 100644 tools/damon/nr_regions.py create mode 100644 tools/damon/record.py create mode 100644 tools/damon/report.py create mode 100644 tools/damon/wss.py diff --git a/tools/damon/.gitignore b/tools/damon/.gitignore new file mode 100644 index 000000000000..96403d36ff93 --- /dev/null +++ b/tools/damon/.gitignore @@ -0,0 +1 @@ +__pycache__/* diff --git a/tools/damon/_dist.py b/tools/damon/_dist.py new file mode 100644 index 000000000000..9851ec964e5c --- /dev/null +++ b/tools/damon/_dist.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import os +import struct +import subprocess + +def access_patterns(f): + nr_regions = struct.unpack('I', f.read(4))[0] + + patterns = [] + for r in range(nr_regions): + saddr = struct.unpack('L', f.read(8))[0] + eaddr = struct.unpack('L', f.read(8))[0] + nr_accesses = struct.unpack('I', f.read(4))[0] + patterns.append([eaddr - saddr, nr_accesses]) + return patterns + +def plot_dist(data_file, output_file, xlabel, ylabel): + terminal = output_file.split('.')[-1] + if not terminal in ['pdf', 'jpeg', 'png', 'svg']: + os.remove(data_file) + print("Unsupported plot output type.") + exit(-1) + + gnuplot_cmd = """ + set term %s; + set output '%s'; + set key off; + set xlabel '%s'; + set ylabel '%s'; + plot '%s' with linespoints;""" % (terminal, output_file, xlabel, ylabel, + data_file) + subprocess.call(['gnuplot', '-e', gnuplot_cmd]) + os.remove(data_file) + diff --git a/tools/damon/_recfile.py b/tools/damon/_recfile.py new file mode 100644 index 000000000000..331b4d8165d8 --- /dev/null +++ b/tools/damon/_recfile.py @@ -0,0 +1,23 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import struct + +fmt_version = 0 + +def set_fmt_version(f): + global fmt_version + + mark = f.read(16) + if mark == b'damon_recfmt_ver': + fmt_version = struct.unpack('i', f.read(4))[0] + else: + fmt_version = 0 + f.seek(0) + return fmt_version + +def pid(f): + if fmt_version == 0: + return struct.unpack('L', f.read(8))[0] + else: + return struct.unpack('i', f.read(4))[0] diff --git a/tools/damon/bin2txt.py b/tools/damon/bin2txt.py new file mode 100644 index 000000000000..8b9b57a0d727 --- /dev/null +++ b/tools/damon/bin2txt.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import argparse +import os +import struct +import sys + +import _recfile + +def parse_time(bindat): + "bindat should be 16 bytes" + sec = struct.unpack('l', bindat[0:8])[0] + nsec = struct.unpack('l', bindat[8:16])[0] + return sec * 1000000000 + nsec; + +def pr_region(f): + saddr = struct.unpack('L', f.read(8))[0] + eaddr = struct.unpack('L', f.read(8))[0] + nr_accesses = struct.unpack('I', f.read(4))[0] + print("%012x-%012x(%10d):\t%d" % + (saddr, eaddr, eaddr - saddr, nr_accesses)) + +def pr_task_info(f): + pid = _recfile.pid(f) + print("pid: ", pid) + nr_regions = struct.unpack('I', f.read(4))[0] + print("nr_regions: ", nr_regions) + for r in range(nr_regions): + pr_region(f) + +def set_argparser(parser): + parser.add_argument('--input', '-i', type=str, metavar='', + default='damon.data', help='input file name') + +def main(args=None): + if not args: + parser = argparse.ArgumentParser() + set_argparser(parser) + args = parser.parse_args() + + file_path = args.input + + if not os.path.isfile(file_path): + print('input file (%s) is not exist' % file_path) + exit(1) + + with open(file_path, 'rb') as f: + _recfile.set_fmt_version(f) + start_time = None + while True: + timebin = f.read(16) + if len(timebin) != 16: + break + time = parse_time(timebin) + if not start_time: + start_time = time + print("start_time: ", start_time) + print("rel time: %16d" % (time - start_time)) + nr_tasks = struct.unpack('I', f.read(4))[0] + print("nr_tasks: ", nr_tasks) + for t in range(nr_tasks): + pr_task_info(f) + print("") + +if __name__ == '__main__': + main() diff --git a/tools/damon/damo b/tools/damon/damo new file mode 100755 index 000000000000..58e1099ae5fc --- /dev/null +++ b/tools/damon/damo @@ -0,0 +1,37 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import argparse + +import record +import report + +class SubCmdHelpFormatter(argparse.RawDescriptionHelpFormatter): + def _format_action(self, action): + parts = super(argparse.RawDescriptionHelpFormatter, + self)._format_action(action) + # skip sub parsers help + if action.nargs == argparse.PARSER: + parts = '\n'.join(parts.split('\n')[1:]) + return parts + +parser = argparse.ArgumentParser(formatter_class=SubCmdHelpFormatter) + +subparser = parser.add_subparsers(title='command', dest='command', + metavar='') +subparser.required = True + +parser_record = subparser.add_parser('record', + help='record data accesses of the given target processes') +record.set_argparser(parser_record) + +parser_report = subparser.add_parser('report', + help='report the recorded data accesses in the specified form') +report.set_argparser(parser_report) + +args = parser.parse_args() + +if args.command == 'record': + record.main(args) +elif args.command == 'report': + report.main(args) diff --git a/tools/damon/heats.py b/tools/damon/heats.py new file mode 100644 index 000000000000..99837083874e --- /dev/null +++ b/tools/damon/heats.py @@ -0,0 +1,362 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +""" +Transform binary trace data into human readable text that can be used for +heatmap drawing, or directly plot the data in a heatmap format. + +Format of the text is: + +