HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
integrated Endpoint (RCiEP) device, providing the capability
to dynamically monitor and tune the PCIe traffic (tune),
and trace the TLP headers (trace).
PTT tune is designed for monitoring and adjusting PCIe link parameters.
We provide several parameters of the PCIe link. Through the driver,
user can adjust the value of certain parameter to affect the PCIe link
for the purpose of enhancing the performance in certian situation.
PTT trace is designed for dumping the TLP headers to the memory, which
can be used to analyze the transactions and usage condition of the PCIe
Link. Users can choose filters to trace headers, by either requester
ID, or those downstream of a set of Root Ports on the same core of the
PTT device. It's also supported to trace the headers of certain type and
of certain direction.
The driver registers a PMU device for each PTT device. The trace can
be used through `perf record` and the traced headers can be decoded
by `perf report`. The perf command support for the device is also
added in this patchset. The tune can be used through the sysfs
attributes of related PMU device. See the documentation for the
detailed usage.
Change since v3:
Address the comments from Jonathan and John, thanks.
- drop members in the common struct which can be get on the fly
- reduce buffer struct and organize the buffers with array instead of list
- reduce the DMA reset wait time to avoid long time busy loop
- split the available_filters sysfs attribute into two files, for root port
and requester respectively. Update the documentation accordingly
- make IOMMU mapping check earlier in probe to avoid race condition. Also
make IOMMU quirk patch prior to driver in the series
- Cleanups and typos fixes from John and Jonathan
Link: https://lore.kernel.org/linux-pci/[email protected]/
Change since v2:
- address the comments from Mathieu, thanks.
- rename the directory to ptt to match the function of the device
- spinoff the declarations to a separate header
- split the trace function to several patches
- some other comments.
- make default smmu domain type of PTT device to identity
Drop the RMR as it's not recommended and use an iommu_def_domain_type
quirk to passthrough the device DMA as suggested by Robin.
Link: https://lore.kernel.org/linux-pci/[email protected]/
Change since v1:
- switch the user interface of trace to perf from debugfs
- switch the user interface of tune to sysfs from debugfs
- add perf tool support to start trace and decode the trace data
- address the comments of documentation from Bjorn
- add RMR[1] support of the device as trace works in RMR mode or
direct DMA mode. RMR support is achieved by common APIs rather
than the APIs implemented in [1].
Link: https://lore.kernel.org/lkml/[email protected]/
[1] https://lore.kernel.org/linux-acpi/[email protected]/
Qi Liu (1):
perf tool: Add support for HiSilicon PCIe Tune and Trace device driver
Yicong Yang (7):
iommu/arm-smmu-v3: Make default domain type of HiSilicon PTT device to
identity
hwtracing: Add trace function support for HiSilicon PCIe Tune and
Trace device
hisi_ptt: Register PMU device for PTT trace
hisi_ptt: Add support for dynamically updating the filter list
hisi_ptt: Add tune function support for HiSilicon PCIe Tune and Trace
device
docs: Add HiSilicon PTT device driver documentation
MAINTAINERS: Add maintainer for HiSilicon PTT driver
Documentation/trace/hisi-ptt.rst | 303 +++++
MAINTAINERS | 7 +
drivers/Makefile | 1 +
drivers/hwtracing/Kconfig | 2 +
drivers/hwtracing/ptt/Kconfig | 11 +
drivers/hwtracing/ptt/Makefile | 2 +
drivers/hwtracing/ptt/hisi_ptt.c | 1133 +++++++++++++++++
drivers/hwtracing/ptt/hisi_ptt.h | 224 ++++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 +
tools/perf/arch/arm/util/auxtrace.c | 56 +-
tools/perf/arch/arm/util/pmu.c | 3 +
tools/perf/arch/arm64/util/Build | 2 +-
tools/perf/arch/arm64/util/hisi_ptt.c | 195 +++
tools/perf/util/Build | 2 +
tools/perf/util/auxtrace.c | 4 +
tools/perf/util/auxtrace.h | 1 +
tools/perf/util/hisi-ptt-decoder/Build | 1 +
.../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c | 170 +++
.../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h | 28 +
tools/perf/util/hisi_ptt.c | 218 ++++
tools/perf/util/hisi_ptt.h | 28 +
21 files changed, 2403 insertions(+), 4 deletions(-)
create mode 100644 Documentation/trace/hisi-ptt.rst
create mode 100644 drivers/hwtracing/ptt/Kconfig
create mode 100644 drivers/hwtracing/ptt/Makefile
create mode 100644 drivers/hwtracing/ptt/hisi_ptt.c
create mode 100644 drivers/hwtracing/ptt/hisi_ptt.h
create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
create mode 100644 tools/perf/util/hisi_ptt.c
create mode 100644 tools/perf/util/hisi_ptt.h
--
2.24.0
From: Qi Liu <[email protected]>
'perf record' and 'perf report --dump-raw-trace' supported in this
patch.
Example usage:
Output will contain raw PTT data and its textual representation, such
as:
0 0 0x5810 [0x30]: PERF_RECORD_AUXTRACE size: 0x400000 offset: 0
ref: 0xa5d50c725 idx: 0 tid: -1 cpu: 0
.
. ... HISI PTT data: size 4194304 bytes
. 00000000: 00 00 00 00 Prefix
. 00000004: 08 20 00 60 Header DW0
. 00000008: ff 02 00 01 Header DW1
. 0000000c: 20 08 00 00 Header DW2
. 00000010: 10 e7 44 ab Header DW3
. 00000014: 2a a8 1e 01 Time
. 00000020: 00 00 00 00 Prefix
. 00000024: 01 00 00 60 Header DW0
. 00000028: 0f 1e 00 01 Header DW1
. 0000002c: 04 00 00 00 Header DW2
. 00000030: 40 00 81 02 Header DW3
. 00000034: ee 02 00 00 Time
....
Signed-off-by: Qi Liu <[email protected]>
Signed-off-by: Yicong Yang <[email protected]>
---
tools/perf/arch/arm/util/auxtrace.c | 56 ++++-
tools/perf/arch/arm/util/pmu.c | 3 +
tools/perf/arch/arm64/util/Build | 2 +-
tools/perf/arch/arm64/util/hisi_ptt.c | 195 ++++++++++++++++
tools/perf/util/Build | 2 +
tools/perf/util/auxtrace.c | 4 +
tools/perf/util/auxtrace.h | 1 +
tools/perf/util/hisi-ptt-decoder/Build | 1 +
.../hisi-ptt-decoder/hisi-ptt-pkt-decoder.c | 170 ++++++++++++++
.../hisi-ptt-decoder/hisi-ptt-pkt-decoder.h | 28 +++
tools/perf/util/hisi_ptt.c | 218 ++++++++++++++++++
tools/perf/util/hisi_ptt.h | 28 +++
12 files changed, 704 insertions(+), 4 deletions(-)
create mode 100644 tools/perf/arch/arm64/util/hisi_ptt.c
create mode 100644 tools/perf/util/hisi-ptt-decoder/Build
create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
create mode 100644 tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
create mode 100644 tools/perf/util/hisi_ptt.c
create mode 100644 tools/perf/util/hisi_ptt.h
diff --git a/tools/perf/arch/arm/util/auxtrace.c b/tools/perf/arch/arm/util/auxtrace.c
index 5fc6a2a3dbc5..dc9d2172464e 100644
--- a/tools/perf/arch/arm/util/auxtrace.c
+++ b/tools/perf/arch/arm/util/auxtrace.c
@@ -4,6 +4,7 @@
* Author: Mathieu Poirier <[email protected]>
*/
+#include <dirent.h>
#include <stdbool.h>
#include <linux/coresight-pmu.h>
#include <linux/zalloc.h>
@@ -14,6 +15,7 @@
#include "../../../util/pmu.h"
#include "cs-etm.h"
#include "arm-spe.h"
+#include "hisi_ptt.h"
static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
{
@@ -50,6 +52,39 @@ static struct perf_pmu **find_all_arm_spe_pmus(int *nr_spes, int *err)
return arm_spe_pmus;
}
+static struct perf_pmu **find_all_hisi_ptt_pmus(int *nr_ptts, int *err)
+{
+ struct perf_pmu **hisi_ptt_pmus = NULL;
+ int nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
+ DIR *dir = NULL;
+ struct dirent *dent;
+
+ hisi_ptt_pmus = zalloc(sizeof(struct perf_pmu *) * nr_cpus);
+ if (!hisi_ptt_pmus) {
+ pr_err("hisi_ptt alloc failed\n");
+ *err = -ENOMEM;
+ return NULL;
+ }
+
+ dir = opendir("/sys/devices");
+ dent = readdir(dir);
+ while (dent) {
+ if (strstr(dent->d_name, HISI_PTT_PMU_NAME)) {
+ hisi_ptt_pmus[*nr_ptts] = perf_pmu__find(dent->d_name);
+ if (hisi_ptt_pmus[*nr_ptts]) {
+ pr_debug2("%s %d: arm_spe_pmu %d type %d name %s\n",
+ __func__, __LINE__, *nr_ptts,
+ hisi_ptt_pmus[*nr_ptts]->type,
+ hisi_ptt_pmus[*nr_ptts]->name);
+ (*nr_ptts)++;
+ }
+ }
+ dent = readdir(dir);
+ }
+
+ return hisi_ptt_pmus;
+}
+
struct auxtrace_record
*auxtrace_record__init(struct evlist *evlist, int *err)
{
@@ -57,8 +92,12 @@ struct auxtrace_record
struct evsel *evsel;
bool found_etm = false;
struct perf_pmu *found_spe = NULL;
+ struct perf_pmu *found_ptt = NULL;
struct perf_pmu **arm_spe_pmus = NULL;
+ struct perf_pmu **hisi_ptt_pmus = NULL;
+
int nr_spes = 0;
+ int nr_ptts = 0;
int i = 0;
if (!evlist)
@@ -66,13 +105,14 @@ struct auxtrace_record
cs_etm_pmu = perf_pmu__find(CORESIGHT_ETM_PMU_NAME);
arm_spe_pmus = find_all_arm_spe_pmus(&nr_spes, err);
+ hisi_ptt_pmus = find_all_hisi_ptt_pmus(&nr_ptts, err);
evlist__for_each_entry(evlist, evsel) {
if (cs_etm_pmu &&
evsel->core.attr.type == cs_etm_pmu->type)
found_etm = true;
- if (!nr_spes || found_spe)
+ if ((!nr_spes || found_spe) && (!nr_ptts || found_ptt))
continue;
for (i = 0; i < nr_spes; i++) {
@@ -81,11 +121,18 @@ struct auxtrace_record
break;
}
}
+
+ for (i = 0; i < nr_ptts; i++) {
+ if (evsel->core.attr.type == hisi_ptt_pmus[i]->type) {
+ found_ptt = hisi_ptt_pmus[i];
+ break;
+ }
+ }
}
free(arm_spe_pmus);
- if (found_etm && found_spe) {
- pr_err("Concurrent ARM Coresight ETM and SPE operation not currently supported\n");
+ if (found_etm && found_spe && found_ptt) {
+ pr_err("Concurrent ARM Coresight ETM ,SPE and HiSilicon PCIe Trace operation not currently supported\n");
*err = -EOPNOTSUPP;
return NULL;
}
@@ -96,6 +143,9 @@ struct auxtrace_record
#if defined(__aarch64__)
if (found_spe)
return arm_spe_recording_init(err, found_spe);
+
+ if (found_ptt)
+ return hisi_ptt_recording_init(err, found_ptt);
#endif
/*
diff --git a/tools/perf/arch/arm/util/pmu.c b/tools/perf/arch/arm/util/pmu.c
index b8b23b9dc598..89a3cedb4557 100644
--- a/tools/perf/arch/arm/util/pmu.c
+++ b/tools/perf/arch/arm/util/pmu.c
@@ -10,6 +10,7 @@
#include <linux/string.h>
#include "arm-spe.h"
+#include "hisi_ptt.h"
#include "../../../util/pmu.h"
struct perf_event_attr
@@ -22,6 +23,8 @@ struct perf_event_attr
#if defined(__aarch64__)
} else if (strstarts(pmu->name, ARM_SPE_PMU_NAME)) {
return arm_spe_pmu_default_config(pmu);
+ } else if (strstarts(pmu->name, HISI_PTT_PMU_NAME)) {
+ pmu->selectable = true;
#endif
}
diff --git a/tools/perf/arch/arm64/util/Build b/tools/perf/arch/arm64/util/Build
index 9fcb4e68add9..8b7fd1dc9f37 100644
--- a/tools/perf/arch/arm64/util/Build
+++ b/tools/perf/arch/arm64/util/Build
@@ -11,4 +11,4 @@ perf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
perf-$(CONFIG_AUXTRACE) += ../../arm/util/pmu.o \
../../arm/util/auxtrace.o \
../../arm/util/cs-etm.o \
- arm-spe.o mem-events.o
+ arm-spe.o mem-events.o hisi_ptt.o
diff --git a/tools/perf/arch/arm64/util/hisi_ptt.c b/tools/perf/arch/arm64/util/hisi_ptt.c
new file mode 100644
index 000000000000..de3a3523ffd5
--- /dev/null
+++ b/tools/perf/arch/arm64/util/hisi_ptt.c
@@ -0,0 +1,195 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/bitops.h>
+#include <linux/log2.h>
+#include <linux/zalloc.h>
+#include <time.h>
+
+#include <internal/lib.h> // page_size
+#include "../../../util/auxtrace.h"
+#include "../../../util/cpumap.h"
+#include "../../../util/debug.h"
+#include "../../../util/event.h"
+#include "../../../util/evlist.h"
+#include "../../../util/evsel.h"
+#include "../../../util/hisi_ptt.h"
+#include "../../../util/pmu.h"
+#include "../../../util/record.h"
+#include "../../../util/session.h"
+#include "../../../util/tsc.h"
+
+#define DEFAULT_PAGE_SIZE 1024
+#define KiB(x) ((x) * DEFAULT_PAGE_SIZE)
+#define MiB(x) ((x) * DEFAULT_PAGE_SIZE * DEFAULT_PAGE_SIZE)
+
+struct hisi_ptt_recording {
+ struct auxtrace_record itr;
+ struct perf_pmu *hisi_ptt_pmu;
+ struct evlist *evlist;
+};
+
+static size_t
+hisi_ptt_info_priv_size(struct auxtrace_record *itr __maybe_unused,
+ struct evlist *evlist __maybe_unused)
+{
+ return HISI_PTT_AUXTRACE_PRIV_SIZE;
+}
+
+static int hisi_ptt_info_fill(struct auxtrace_record *itr,
+ struct perf_session *session,
+ struct perf_record_auxtrace_info *auxtrace_info,
+ size_t priv_size)
+{
+ struct hisi_ptt_recording *pttr =
+ container_of(itr, struct hisi_ptt_recording, itr);
+ struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
+
+ if (priv_size != HISI_PTT_AUXTRACE_PRIV_SIZE)
+ return -EINVAL;
+
+ if (!session->evlist->core.nr_mmaps)
+ return -EINVAL;
+
+ auxtrace_info->type = PERF_AUXTRACE_HISI_PTT;
+ auxtrace_info->priv[HISI_PTT_PMU_TYPE] = hisi_ptt_pmu->type;
+
+ return 0;
+}
+
+static int hisi_ptt_set_auxtrace_mmap_page(struct record_opts *opts)
+{
+ bool privileged = perf_event_paranoid_check(-1);
+
+ if (!opts->full_auxtrace)
+ return 0;
+
+ if (opts->full_auxtrace && !opts->auxtrace_mmap_pages) {
+ if (privileged) {
+ opts->auxtrace_mmap_pages = MiB(16) / page_size;
+ } else {
+ opts->auxtrace_mmap_pages = KiB(128) / page_size;
+ if (opts->mmap_pages == UINT_MAX)
+ opts->mmap_pages = KiB(256) / page_size;
+ }
+ }
+
+ /* Validate auxtrace_mmap_pages */
+ if (opts->auxtrace_mmap_pages) {
+ size_t sz = opts->auxtrace_mmap_pages * (size_t)page_size;
+ size_t min_sz = KiB(8);
+
+ if (sz < min_sz || !is_power_of_2(sz)) {
+ pr_err("Invalid mmap size for HISI PTT: must be at least %zuKiB and a power of 2\n",
+ min_sz / DEFAULT_PAGE_SIZE);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+static int hisi_ptt_recording_options(struct auxtrace_record *itr,
+ struct evlist *evlist,
+ struct record_opts *opts)
+{
+ struct hisi_ptt_recording *pttr =
+ container_of(itr, struct hisi_ptt_recording, itr);
+ struct perf_pmu *hisi_ptt_pmu = pttr->hisi_ptt_pmu;
+ struct perf_cpu_map *cpus = evlist->core.cpus;
+ struct evsel *evsel, *hisi_ptt_evsel = NULL;
+ struct evsel *tracking_evsel;
+ int err;
+
+ pttr->evlist = evlist;
+ evlist__for_each_entry(evlist, evsel) {
+ if (evsel->core.attr.type == hisi_ptt_pmu->type) {
+ if (hisi_ptt_evsel) {
+ pr_err("There may be only one " HISI_PTT_PMU_NAME "x event\n");
+ return -EINVAL;
+ }
+ evsel->core.attr.freq = 0;
+ evsel->core.attr.sample_period = 1;
+ hisi_ptt_evsel = evsel;
+ opts->full_auxtrace = true;
+ }
+ }
+
+ err = hisi_ptt_set_auxtrace_mmap_page(opts);
+ if (err)
+ return err;
+ /*
+ * To obtain the auxtrace buffer file descriptor, the auxtrace event
+ * must come first.
+ */
+ evlist__to_front(evlist, hisi_ptt_evsel);
+
+ if (!perf_cpu_map__empty(cpus)) {
+ evsel__set_sample_bit(hisi_ptt_evsel, TIME);
+ evsel__set_sample_bit(hisi_ptt_evsel, CPU);
+ }
+
+ /* Add dummy event to keep tracking */
+ err = parse_events(evlist, "dummy:u", NULL);
+ if (err)
+ return err;
+
+ tracking_evsel = evlist__last(evlist);
+ evlist__set_tracking_event(evlist, tracking_evsel);
+
+ tracking_evsel->core.attr.freq = 0;
+ tracking_evsel->core.attr.sample_period = 1;
+
+ if (!perf_cpu_map__empty(cpus))
+ evsel__set_sample_bit(tracking_evsel, TIME);
+
+ return 0;
+}
+
+static u64 hisi_ptt_reference(struct auxtrace_record *itr __maybe_unused)
+{
+ return rdtsc();
+}
+
+static void hisi_ptt_recording_free(struct auxtrace_record *itr)
+{
+ struct hisi_ptt_recording *pttr =
+ container_of(itr, struct hisi_ptt_recording, itr);
+
+ free(pttr);
+}
+
+struct auxtrace_record *hisi_ptt_recording_init(int *err,
+ struct perf_pmu *hisi_ptt_pmu)
+{
+ struct hisi_ptt_recording *pttr;
+
+ if (!hisi_ptt_pmu) {
+ *err = -ENODEV;
+ return NULL;
+ }
+
+ pttr = zalloc(sizeof(struct hisi_ptt_recording));
+ if (!pttr) {
+ *err = -ENOMEM;
+ return NULL;
+ }
+
+ pttr->hisi_ptt_pmu = hisi_ptt_pmu;
+ pttr->itr.pmu = hisi_ptt_pmu;
+ pttr->itr.recording_options = hisi_ptt_recording_options;
+ pttr->itr.info_priv_size = hisi_ptt_info_priv_size;
+ pttr->itr.info_fill = hisi_ptt_info_fill;
+ pttr->itr.free = hisi_ptt_recording_free;
+ pttr->itr.reference = hisi_ptt_reference;
+ pttr->itr.read_finish = auxtrace_record__read_finish;
+ pttr->itr.alignment = 0;
+
+ *err = 0;
+ return &pttr->itr;
+}
diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index 2a403cefcaf2..a32d1ef8ce29 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -115,6 +115,8 @@ perf-$(CONFIG_AUXTRACE) += intel-pt.o
perf-$(CONFIG_AUXTRACE) += intel-bts.o
perf-$(CONFIG_AUXTRACE) += arm-spe.o
perf-$(CONFIG_AUXTRACE) += arm-spe-decoder/
+perf-$(CONFIG_AUXTRACE) += hisi_ptt.o
+perf-$(CONFIG_AUXTRACE) += hisi-ptt-decoder/
perf-$(CONFIG_AUXTRACE) += s390-cpumsf.o
ifdef CONFIG_LIBOPENCSD
diff --git a/tools/perf/util/auxtrace.c b/tools/perf/util/auxtrace.c
index 825336304a37..7f1ee06538a0 100644
--- a/tools/perf/util/auxtrace.c
+++ b/tools/perf/util/auxtrace.c
@@ -51,6 +51,7 @@
#include "intel-pt.h"
#include "intel-bts.h"
#include "arm-spe.h"
+#include "hisi_ptt.h"
#include "s390-cpumsf.h"
#include "util/mmap.h"
@@ -1281,6 +1282,9 @@ int perf_event__process_auxtrace_info(struct perf_session *session,
case PERF_AUXTRACE_S390_CPUMSF:
err = s390_cpumsf_process_auxtrace_info(event, session);
break;
+ case PERF_AUXTRACE_HISI_PTT:
+ err = hisi_ptt_process_auxtrace_info(event, session);
+ break;
case PERF_AUXTRACE_UNKNOWN:
default:
return -EINVAL;
diff --git a/tools/perf/util/auxtrace.h b/tools/perf/util/auxtrace.h
index 19910b9011f3..0f7eaad4d8d0 100644
--- a/tools/perf/util/auxtrace.h
+++ b/tools/perf/util/auxtrace.h
@@ -48,6 +48,7 @@ enum auxtrace_type {
PERF_AUXTRACE_CS_ETM,
PERF_AUXTRACE_ARM_SPE,
PERF_AUXTRACE_S390_CPUMSF,
+ PERF_AUXTRACE_HISI_PTT,
};
enum itrace_period_type {
diff --git a/tools/perf/util/hisi-ptt-decoder/Build b/tools/perf/util/hisi-ptt-decoder/Build
new file mode 100644
index 000000000000..db3db8b75033
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/Build
@@ -0,0 +1 @@
+perf-$(CONFIG_AUXTRACE) += hisi-ptt-pkt-decoder.o
diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
new file mode 100644
index 000000000000..3fb7a6949209
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.c
@@ -0,0 +1,170 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <endian.h>
+#include <byteswap.h>
+#include <linux/bitops.h>
+#include <stdarg.h>
+
+#include "../color.h"
+#include "hisi-ptt-pkt-decoder.h"
+
+/*
+ * For 8DW format, the bit[31:11] of DW0 is always 0x1fffff, which can be
+ * used to distinguish the data format.
+ * 8DW format is like:
+ * bits [ 31:11 ][ 10:0 ]
+ * |---------------------------------------|-------------------|
+ * DW0 [ 0x1fffff ][ Reserved (0x7ff) ]
+ * DW1 [ Prefix ]
+ * DW2 [ Header DW0 ]
+ * DW3 [ Header DW1 ]
+ * DW4 [ Header DW2 ]
+ * DW5 [ Header DW3 ]
+ * DW6 [ Reserved (0x0) ]
+ * DW7 [ Time ]
+ *
+ * 4DW format is like:
+ * bits [31:30] [ 29:25 ][24][23][22][21][ 20:11 ][ 10:0 ]
+ * |-----|---------|---|---|---|---|-------------|-------------|
+ * DW0 [ Fmt ][ Type ][T9][T8][TH][SO][ Length ][ Time ]
+ * DW1 [ Header DW1 ]
+ * DW2 [ Header DW2 ]
+ * DW3 [ Header DW3 ]
+ */
+
+enum hisi_ptt_8dw_pkt_field_type {
+ HISI_PTT_8DW_PREFIX,
+ HISI_PTT_8DW_HEAD0,
+ HISI_PTT_8DW_HEAD1,
+ HISI_PTT_8DW_HEAD2,
+ HISI_PTT_8DW_HEAD3,
+ HISI_PTT_8DW_TIME,
+ HISI_PTT_8DW_TYPE_MAX
+};
+
+enum hisi_ptt_4dw_pkt_field_type {
+ HISI_PTT_4DW_HEAD1,
+ HISI_PTT_4DW_HEAD2,
+ HISI_PTT_4DW_HEAD3,
+ HISI_PTT_4DW_TYPE_MAX
+};
+
+static const char * const hisi_ptt_8dw_pkt_field_name[] = {
+ [HISI_PTT_8DW_PREFIX] = "Prefix",
+ [HISI_PTT_8DW_HEAD0] = "Header DW0",
+ [HISI_PTT_8DW_HEAD1] = "Header DW1",
+ [HISI_PTT_8DW_HEAD2] = "Header DW2",
+ [HISI_PTT_8DW_HEAD3] = "Header DW3",
+ [HISI_PTT_8DW_TIME] = "Time",
+};
+
+static const char * const hisi_ptt_4dw_pkt_field_name[] = {
+ [HISI_PTT_4DW_HEAD1] = "Header DW1",
+ [HISI_PTT_4DW_HEAD2] = "Header DW2",
+ [HISI_PTT_4DW_HEAD3] = "Header DW3",
+};
+
+/* offset of each member is determined by format of 8dw packet. */
+static uint32_t hisi_ptt_8dw_pkt_field_offset[] = {
+ [HISI_PTT_8DW_PREFIX] = 4,
+ [HISI_PTT_8DW_HEAD0] = 4,
+ [HISI_PTT_8DW_HEAD1] = 4,
+ [HISI_PTT_8DW_HEAD2] = 4,
+ [HISI_PTT_8DW_HEAD3] = 4,
+ [HISI_PTT_8DW_TIME] = 8,
+};
+
+union hisi_ptt_4dw {
+ struct {
+ uint32_t format : 2;
+ uint32_t type : 5;
+ uint32_t t9 : 1;
+ uint32_t t8 : 1;
+ uint32_t th : 1;
+ uint32_t so : 1;
+ uint32_t len : 10;
+ uint32_t time : 11;
+ };
+ uint32_t value;
+};
+
+static void hisi_ptt_print_pkt(const unsigned char *buf, int *pos, const char *desc)
+{
+ const char *color = PERF_COLOR_BLUE;
+ int field_len = sizeof(uint32_t);
+ int i;
+
+ printf(".");
+ color_fprintf(stdout, color, " %08x: ", *pos);
+ for (i = 0; i < field_len; i++)
+ color_fprintf(stdout, color, "%02x ", buf[i]);
+ for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
+ color_fprintf(stdout, color, " ");
+ color_fprintf(stdout, color, " %s\n", desc);
+ *pos += field_len;
+}
+
+static int hisi_ptt_8dw_kpt_desc(const unsigned char *buf, int pos)
+{
+ int i;
+
+ for (i = HISI_PTT_8DW_PREFIX; i < HISI_PTT_8DW_TYPE_MAX; i++) {
+ buf += hisi_ptt_8dw_pkt_field_offset[i];
+ hisi_ptt_print_pkt(buf, &pos, hisi_ptt_8dw_pkt_field_name[i]);
+ }
+
+ return HISI_PTT_8DW_PKT_SIZE;
+}
+
+static void hisi_ptt_4dw_print_dw0(const unsigned char *buf, int *pos)
+{
+ const char *color = PERF_COLOR_BLUE;
+ int field_len = sizeof(uint32_t);
+ union hisi_ptt_4dw dw0;
+ int i;
+
+ dw0.value = *(uint32_t *)buf;
+ printf(".");
+ color_fprintf(stdout, color, " %08x: ", *pos);
+ for (i = 0; i < field_len; i++)
+ color_fprintf(stdout, color, "%02x ", buf[i]);
+ for (i = 0; i < HISI_PTT_MAX_SPACE_LEN; i++)
+ color_fprintf(stdout, color, " ");
+
+ color_fprintf(stdout, color,
+ " %s %x %s %x %s %x %s %x %s %x %s %x %s %x %s %x\n",
+ "Format", dw0.format, "Type", dw0.type, "T9", dw0.t9,
+ "T8", dw0.t8, "TH", dw0.th, "SO", dw0.so, "Length",
+ dw0.len, "Time", dw0.time);
+
+ *pos += field_len;
+}
+
+static int hisi_ptt_4dw_kpt_desc(const unsigned char *buf, int pos)
+{
+ int i;
+
+ hisi_ptt_4dw_print_dw0(buf, &pos);
+
+ for (i = HISI_PTT_4DW_HEAD1; i < HISI_PTT_4DW_TYPE_MAX; i++) {
+ buf += sizeof(uint32_t);
+ hisi_ptt_print_pkt(buf, &pos, hisi_ptt_4dw_pkt_field_name[i]);
+ }
+
+ return HISI_PTT_4DW_PKT_SIZE;
+}
+
+int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type)
+{
+ if (type == HISI_PTT_8DW_PKT)
+ return hisi_ptt_8dw_kpt_desc(buf, pos);
+
+ return hisi_ptt_4dw_kpt_desc(buf, pos);
+}
diff --git a/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
new file mode 100644
index 000000000000..20c51f230cac
--- /dev/null
+++ b/tools/perf/util/hisi-ptt-decoder/hisi-ptt-pkt-decoder.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#ifndef INCLUDE__HISI_PTT_PKT_DECODER_H__
+#define INCLUDE__HISI_PTT_PKT_DECODER_H__
+
+#include <stddef.h>
+#include <stdint.h>
+
+#define HISI_PTT_PKT_DESC_MAX 256
+#define HISI_PTT_NEED_MORE_BYTES -1
+#define HISI_PTT_8DW_CHECK_MASK GENMASK(31, 11)
+#define HISI_PTT_IS_8DW_PKT GENMASK(31, 11)
+#define HISI_PTT_8DW_PKT_SIZE 32
+#define HISI_PTT_4DW_PKT_SIZE 16
+#define HISI_PTT_MAX_SPACE_LEN 10
+
+enum hisi_ptt_pkt_type {
+ HISI_PTT_4DW_PKT,
+ HISI_PTT_8DW_PKT,
+};
+
+int hisi_ptt_pkt_desc(const unsigned char *buf, int pos, enum hisi_ptt_pkt_type type);
+
+#endif
diff --git a/tools/perf/util/hisi_ptt.c b/tools/perf/util/hisi_ptt.c
new file mode 100644
index 000000000000..e2934214763c
--- /dev/null
+++ b/tools/perf/util/hisi_ptt.c
@@ -0,0 +1,218 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#include <byteswap.h>
+#include <endian.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <linux/bitops.h>
+#include <linux/kernel.h>
+#include <linux/log2.h>
+#include <linux/types.h>
+#include <linux/zalloc.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include "auxtrace.h"
+#include "color.h"
+#include "debug.h"
+#include "evlist.h"
+#include "evsel.h"
+#include "hisi_ptt.h"
+#include "hisi-ptt-decoder/hisi-ptt-pkt-decoder.h"
+#include "machine.h"
+#include "session.h"
+#include "symbol.h"
+#include "tool.h"
+#include "util/synthetic-events.h"
+
+struct hisi_ptt {
+ struct auxtrace auxtrace;
+ struct auxtrace_queues queues;
+ u32 auxtrace_type;
+ struct perf_session *session;
+ struct machine *machine;
+ u32 pmu_type;
+};
+
+struct hisi_ptt_queue {
+ struct hisi_ptt *ptt;
+ struct auxtrace_buffer *buffer;
+};
+
+static enum hisi_ptt_pkt_type hisi_ptt_check_packet_type(unsigned char *buf)
+{
+ uint32_t head = *(uint32_t *)buf;
+
+ if ((HISI_PTT_8DW_CHECK_MASK & head) == HISI_PTT_IS_8DW_PKT)
+ return HISI_PTT_8DW_PKT;
+
+ return HISI_PTT_4DW_PKT;
+}
+
+static void hisi_ptt_dump(struct hisi_ptt *ptt __maybe_unused,
+ unsigned char *buf, size_t len)
+{
+ const char *color = PERF_COLOR_BLUE;
+ enum hisi_ptt_pkt_type type;
+ size_t pos = 0;
+ int pkt_len;
+
+ color_fprintf(stdout, color, ". ... HISI PTT data: size %zu bytes\n",
+ len);
+
+ type = hisi_ptt_check_packet_type(buf);
+ while (len) {
+ pkt_len = hisi_ptt_pkt_desc(buf, pos, type);
+ if (!pkt_len)
+ color_fprintf(stdout, color, " Bad packet!\n");
+
+ pos += pkt_len;
+ buf += pkt_len;
+ len -= pkt_len;
+ }
+}
+
+static void hisi_ptt_dump_event(struct hisi_ptt *ptt, unsigned char *buf,
+ size_t len)
+{
+ printf(".\n");
+
+ hisi_ptt_dump(ptt, buf, len);
+}
+
+static int hisi_ptt_process_event(struct perf_session *session __maybe_unused,
+ union perf_event *event __maybe_unused,
+ struct perf_sample *sample __maybe_unused,
+ struct perf_tool *tool __maybe_unused)
+{
+ return 0;
+}
+
+static int hisi_ptt_process_auxtrace_event(struct perf_session *session,
+ union perf_event *event,
+ struct perf_tool *tool __maybe_unused)
+{
+ struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+ auxtrace);
+ struct auxtrace_buffer *buffer;
+ off_t data_offset;
+ int fd = perf_data__fd(session->data);
+ int err;
+
+ if (perf_data__is_pipe(session->data)) {
+ data_offset = 0;
+ } else {
+ data_offset = lseek(fd, 0, SEEK_CUR);
+ if (data_offset == -1)
+ return -errno;
+ }
+
+ err = auxtrace_queues__add_event(&ptt->queues, session, event,
+ data_offset, &buffer);
+ if (err)
+ return err;
+
+ if (dump_trace) {
+ if (auxtrace_buffer__get_data(buffer, fd)) {
+ hisi_ptt_dump_event(ptt, buffer->data, buffer->size);
+ auxtrace_buffer__put_data(buffer);
+ }
+ }
+
+ return 0;
+}
+
+static int hisi_ptt_flush(struct perf_session *session __maybe_unused,
+ struct perf_tool *tool __maybe_unused)
+{
+ return 0;
+}
+
+static void hisi_ptt_free_events(struct perf_session *session)
+{
+ struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+ auxtrace);
+ struct auxtrace_queues *queues = &ptt->queues;
+ unsigned int i;
+
+ for (i = 0; i < queues->nr_queues; i++) {
+ free(queues->queue_array[i].priv);
+ queues->queue_array[i].priv = NULL;
+ }
+ auxtrace_queues__free(queues);
+}
+
+static void hisi_ptt_free(struct perf_session *session)
+{
+ struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt,
+ auxtrace);
+
+ hisi_ptt_free_events(session);
+ session->auxtrace = NULL;
+ free(ptt);
+}
+
+static bool hisi_ptt_evsel_is_auxtrace(struct perf_session *session,
+ struct evsel *evsel)
+{
+ struct hisi_ptt *ptt = container_of(session->auxtrace, struct hisi_ptt, auxtrace);
+
+ return evsel->core.attr.type == ptt->pmu_type;
+}
+
+static const char * const hisi_ptt_info_fmts[] = {
+ [HISI_PTT_PMU_TYPE] = " PMU Type %" PRId64 "\n",
+};
+
+static void hisi_ptt_print_info(__u64 *arr)
+{
+ if (!dump_trace)
+ return;
+
+ fprintf(stdout, hisi_ptt_info_fmts[HISI_PTT_PMU_TYPE], arr[HISI_PTT_PMU_TYPE]);
+}
+
+int hisi_ptt_process_auxtrace_info(union perf_event *event,
+ struct perf_session *session)
+{
+ struct perf_record_auxtrace_info *auxtrace_info = &event->auxtrace_info;
+ struct hisi_ptt *ptt;
+ int err;
+
+ if (auxtrace_info->header.size < HISI_PTT_AUXTRACE_PRIV_SIZE +
+ sizeof(struct perf_record_auxtrace_info))
+ return -EINVAL;
+
+ ptt = zalloc(sizeof(struct hisi_ptt));
+ if (!ptt)
+ return -ENOMEM;
+
+ err = auxtrace_queues__init(&ptt->queues);
+ if (err)
+ goto err_free;
+
+ ptt->session = session;
+ ptt->machine = &session->machines.host; /* No kvm support */
+ ptt->auxtrace_type = auxtrace_info->type;
+ ptt->pmu_type = auxtrace_info->priv[HISI_PTT_PMU_TYPE];
+
+ ptt->auxtrace.process_event = hisi_ptt_process_event;
+ ptt->auxtrace.process_auxtrace_event = hisi_ptt_process_auxtrace_event;
+ ptt->auxtrace.flush_events = hisi_ptt_flush;
+ ptt->auxtrace.free_events = hisi_ptt_free_events;
+ ptt->auxtrace.free = hisi_ptt_free;
+ ptt->auxtrace.evsel_is_auxtrace = hisi_ptt_evsel_is_auxtrace;
+ session->auxtrace = &ptt->auxtrace;
+
+ hisi_ptt_print_info(&auxtrace_info->priv[0]);
+
+ return 0;
+
+err_free:
+ free(ptt);
+ return err;
+}
diff --git a/tools/perf/util/hisi_ptt.h b/tools/perf/util/hisi_ptt.h
new file mode 100644
index 000000000000..c0b6cbde1221
--- /dev/null
+++ b/tools/perf/util/hisi_ptt.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * HiSilicon PCIe Trace and Tuning (PTT) support
+ * Copyright (c) 2022 HiSilicon Technologies Co., Ltd.
+ */
+
+#ifndef INCLUDE__PERF_HISI_PTT_H__
+#define INCLUDE__PERF_HISI_PTT_H__
+
+#define HISI_PTT_PMU_NAME "hisi_ptt"
+enum {
+ HISI_PTT_PMU_TYPE,
+ HISI_PTT_PER_CPU_MMAPS,
+ HISI_PTT_AUXTRACE_PRIV_MAX,
+};
+
+#define HISI_PTT_AUXTRACE_PRIV_SIZE (HISI_PTT_AUXTRACE_PRIV_MAX * sizeof(u64))
+union perf_event;
+struct perf_session;
+struct perf_pmu;
+
+struct auxtrace_record *hisi_ptt_recording_init(int *err,
+ struct perf_pmu *hisi_ptt_pmu);
+
+int hisi_ptt_process_auxtrace_info(union perf_event *event,
+ struct perf_session *session);
+
+#endif
--
2.24.0
Document the introduction and usage of HiSilicon PTT device driver.
Signed-off-by: Yicong Yang <[email protected]>
---
Documentation/trace/hisi-ptt.rst | 303 +++++++++++++++++++++++++++++++
1 file changed, 303 insertions(+)
create mode 100644 Documentation/trace/hisi-ptt.rst
diff --git a/Documentation/trace/hisi-ptt.rst b/Documentation/trace/hisi-ptt.rst
new file mode 100644
index 000000000000..13677705ee1f
--- /dev/null
+++ b/Documentation/trace/hisi-ptt.rst
@@ -0,0 +1,303 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+======================================
+HiSilicon PCIe Tune and Trace device
+======================================
+
+Introduction
+============
+
+HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
+integrated Endpoint (RCiEP) device, providing the capability
+to dynamically monitor and tune the PCIe link's events (tune),
+and trace the TLP headers (trace). The two functions are independent,
+but is recommended to use them together to analyze and enhance the
+PCIe link's performance.
+
+On Kunpeng 930 SoC, the PCIe Root Complex is composed of several
+PCIe cores. Each PCIe core includes several Root Ports and a PTT
+RCiEP, like below. The PTT device is capable of tuning and
+tracing the links of the PCIe core.
+::
+ +--------------Core 0-------+
+ | | [ PTT ] |
+ | | [Root Port]---[Endpoint]
+ | | [Root Port]---[Endpoint]
+ | | [Root Port]---[Endpoint]
+ Root Complex |------Core 1-------+
+ | | [ PTT ] |
+ | | [Root Port]---[ Switch ]---[Endpoint]
+ | | [Root Port]---[Endpoint] `-[Endpoint]
+ | | [Root Port]---[Endpoint]
+ +---------------------------+
+
+The PTT device driver registers one PMU device for each PTT device.
+The name of each PTT device is composed of 'hisi_ptt' prefix with
+the id of the SICL and the Core where it locates. The Kunpeng 930
+SoC encapsulates multiple CPU dies (SCCL, Super CPU Cluster) and
+IO dies (SICL, Super I/O Cluster), where there's one PCIe Root
+Complex for each SICL.
+::
+ /sys/devices/hisi_ptt<sicl_id>_<core_id>
+
+Tune
+====
+
+PTT tune is designed for monitoring and adjusting PCIe link parameters (events).
+Currently we support events in 4 classes. The scope of the events
+covers the PCIe core to which the PTT device belongs.
+
+Each event is presented as a file under $(PTT PMU dir)/tune, and
+a simple open/read/write/close cycle will be used to tune the event.
+::
+ $ cd /sys/devices/hisi_ptt<sicl_id>_<core_id>/tune
+ $ ls
+ qos_tx_cpl qos_tx_np qos_tx_p
+ tx_path_rx_req_alloc_buf_level
+ tx_path_tx_req_alloc_buf_level
+ $ cat qos_tx_dp
+ 1
+ $ echo 2 > qos_tx_dp
+ $ cat qos_tx_dp
+ 2
+
+Current value (numerical value) of the event can be simply read
+from the file, and the desired value written to the file to tune.
+
+1. Tx path QoS control
+------------------------
+
+The following files are provided to tune the QoS of the tx path of
+the PCIe core.
+
+- qos_tx_cpl: weight of Tx completion TLPs
+- qos_tx_np: weight of Tx non-posted TLPs
+- qos_tx_p: weight of Tx posted TLPs
+
+The weight influences the proportion of certain packets on the PCIe link.
+For example, for the storage scenario, increase the proportion
+of the completion packets on the link to enhance the performance as
+more completions are consumed.
+
+The available tune data of these events is [0, 1, 2].
+Writing a negative value will return an error, and out of range
+values will be converted to 2. Note that the event value just
+indicates a probable level, but is not precise.
+
+2. Tx path buffer control
+-------------------------
+
+Following files are provided to tune the buffer of tx path of the PCIe core.
+
+- tx_path_rx_req_alloc_buf_level: watermark of Rx requested
+- tx_path_tx_req_alloc_buf_level: watermark of Tx requested
+
+These events influence the watermark of the buffer allocated for each
+type. Rx means the inbound while Tx means outbound. The packets will
+be stored in the buffer first and then transmitted either when the
+watermark reached or when timed out. For a busy direction, you should
+increase the related buffer watermark to avoid frequently posting and
+thus enhance the performance. In most cases just keep the default value.
+
+The available tune data of above events is [0, 1, 2].
+Writing a negative value will return an error, and out of range
+values will be converted to 2. Note that the event value just
+indicates a probable level, but is not precise.
+
+Trace
+=====
+
+PTT trace is designed for dumping the TLP headers to the memory, which
+can be used to analyze the transactions and usage condition of the PCIe
+Link. You can choose to filter the traced headers by either requester ID,
+or those downstream of a set of Root Ports on the same core of the PTT
+device. It's also supported to trace the headers of certain type and of
+certain direction.
+
+You can use the perf command `perf record` to set the parameters, start
+trace and get the data. It's also supported to decode the trace
+data with `perf report`. The control parameters for trace is inputted
+as event code for each events, which will be further illustrated later.
+An example usage is like
+::
+ $ perf record -e hisi_ptt0_2/filter=0x80001,type=1,direction=1,
+ format=1/ -- sleep 5
+
+This will trace the TLP headers downstream root port 0000:00:10.1 (event
+code for event 'filter' is 0x80001) with type of posted TLP requests,
+direction of inbound and traced data format of 8DW.
+
+1. filter
+---------
+
+The TLP headers to trace can be filtered by the Root Ports or the requester
+ID of the endpoints, which are located on the same core of the PTT device.
+You can set the filter by specifying the `filter` parameter which is required
+to start the trace. The parameter value is 20 bit. The supported filters and
+related values are outputted through `available_root_port_filters` and
+`available_requester_filters` sysfs attributes for Root Ports and Requesters
+respectively.
+::
+ $ cat available_root_port_filters
+ 0000:00:10.0 0x80001
+ 0000:00:11.0 0x80004
+ $ cat available_requester_filters
+ 0000:01:00.0 0x00100
+ 0000:01:00.1 0x00101
+
+Note that multiple Root Ports can be specified at one time, but only
+one Endpoint function can be specified in one trace. Specifying both
+Root Port and function at the same time is not supported.
+
+If no filter is available, reading the related filter sysfs attribute
+will get an empty string.
+::
+ $ cat available_root_port_filters
+
+ $ cat available_requester_filters
+
+The available filters can be dynamically updated, which means you can always
+get correct filter information when hotplug events happen, or when you manually
+remove/rescan the devices.
+
+2. type
+-------
+
+You can trace the TLP headers of certain types by specifying the `type`
+parameter, which is required to start the trace. The parameter value is
+8 bit. Current supported types and related values are shown below:
+
+8'b00000001: posted requests (P)
+8'b00000010: non-posted requests (NP)
+8'b00000100: completions (CPL)
+
+You can specify multiple types when tracing inbound TLP headers, but can only
+specify one when tracing outbound TLP headers.
+
+3. direction
+------------
+
+You can trace the TLP headers from certain direction, which is relative
+to the Root Port or the PCIe core, by specifying the `direction` parameter.
+This is optional and the default parameter is inbound. The parameter value
+is 4 bit. When the desired format is 4DW, directions and related values
+supported are shown below:
+
+4'b0000: inbound TLPs (P, NP, CPL)
+4'b0001: outbound TLPs (P, NP, CPL)
+4'b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B)
+4'b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A)
+
+When the desired format is 8DW, directions and related values supported are
+shown below:
+
+4'b0000: reserved
+4'b0001: outbound TLPs (P, NP, CPL)
+4'b0010: inbound TLPs (P, NP, CPL B)
+4'b0011: inbound TLPs (CPL A)
+
+Inbound completions are classified into two types:
+
+completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B
+completion B (CPL B): completion of DMA remote2local and P2P non-posted requests
+
+4. format
+--------------
+
+You can change the format of the traced TLP headers by specifying the
+`format` parameter. The default format is 4DW. The parameter value is 4 bit.
+Current supported formats and related values are shown below:
+
+4'b0000: 4DW length per TLP header
+4'b0001: 8DW length per TLP header
+
+The traced TLP header format is different from the PCIe standard.
+
+When using the 8DW data format, the entire TLP header is logged
+(Header DW0-3 shown below). For example, the TLP header for Memory
+Reads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17;
+the header for Configuration Requests is shown in Figure 2.20, etc.
+
+In addition, 8DW trace buffer entries contain a timestamp and
+possibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0).
+Otherwise this field will be all 0.
+
+The bit[31:11] of DW0 is always 0x1fffff, which can be
+used to distinguish the data format. 8DW format is like
+::
+ bits [ 31:11 ][ 10:0 ]
+ |---------------------------------------|-------------------|
+ DW0 [ 0x1fffff ][ Reserved (0x7ff) ]
+ DW1 [ Prefix ]
+ DW2 [ Header DW0 ]
+ DW3 [ Header DW1 ]
+ DW4 [ Header DW2 ]
+ DW5 [ Header DW3 ]
+ DW6 [ Reserved (0x0) ]
+ DW7 [ Time ]
+
+When using the 4DW data format, DW0 of the trace buffer entry
+contains selected fields of DW0 of the TLP, together with a
+timestamp. DW1-DW3 of the trace buffer entry contain DW1-DW3
+directly from the TLP header.
+
+4DW format is like
+::
+ bits [31:30] [ 29:25 ][24][23][22][21][ 20:11 ][ 10:0 ]
+ |-----|---------|---|---|---|---|-------------|-------------|
+ DW0 [ Fmt ][ Type ][T9][T8][TH][SO][ Length ][ Time ]
+ DW1 [ Header DW1 ]
+ DW2 [ Header DW2 ]
+ DW3 [ Header DW3 ]
+
+5. memory management
+--------------------
+
+The traced TLP headers will be written to the memory allocated
+by the driver. The hardware accepts 4 DMA address with same size,
+and writes the buffer sequentially like below. If DMA addr 3 is
+finished and the trace is still on, it will return to addr 0.
+::
+ +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-+
+ +---------------------------------------------------------+
+
+Driver will allocate each DMA buffer of 4MiB. The finished buffer
+will be copied to the perf AUX buffer allocated by the perf core.
+Once the AUX buffer is full while the trace is still on, driver
+will commit the AUX buffer first and then apply for a new one with
+the same size. The size of AUX buffer is default to 16MiB. User can
+adjust the size by specifying the `-m` parameter of the perf command.
+
+Note that there is a gap between committing the old AUX buffer and
+applying a new one, which means the trace is stopped during the
+moment and TLPs transferred in the moment cannot be traced. To avoid
+this situation, you should begin the trace with large AUX buffer
+enough to avoid this gap.
+
+6. decoding
+-----------
+
+You can decode the traced data with `perf report -D` command (currently
+only support to dump the raw trace data). The traced data will be decoded
+according to the format described previously (take 8DW as an example):
+::
+ [...perf headers and other information]
+ . ... HISI PTT data: size 4194304 bytes
+ . 00000000: 00 00 00 00 Prefix
+ . 00000004: 01 00 00 60 Header DW0
+ . 00000008: 0f 1e 00 01 Header DW1
+ . 0000000c: 04 00 00 00 Header DW2
+ . 00000010: 40 00 81 02 Header DW3
+ . 00000014: 33 c0 04 00 Time
+ . 00000020: 00 00 00 00 Prefix
+ . 00000024: 01 00 00 60 Header DW0
+ . 00000028: 0f 1e 00 01 Header DW1
+ . 0000002c: 04 00 00 00 Header DW2
+ . 00000030: 40 00 81 02 Header DW3
+ . 00000034: 02 00 00 00 Time
+ . 00000040: 00 00 00 00 Prefix
+ . 00000044: 01 00 00 60 Header DW0
+ . 00000048: 0f 1e 00 01 Header DW1
+ . 0000004c: 04 00 00 00 Header DW2
+ . 00000050: 40 00 81 02 Header DW3
+ [...]
--
2.24.0
On Mon, 21 Feb 2022 16:43:06 +0800
Yicong Yang <[email protected]> wrote:
> Document the introduction and usage of HiSilicon PTT device driver.
>
> Signed-off-by: Yicong Yang <[email protected]>
Reviewed-by: Jonathan Cameron <[email protected]>
> ---
> Documentation/trace/hisi-ptt.rst | 303 +++++++++++++++++++++++++++++++
> 1 file changed, 303 insertions(+)
> create mode 100644 Documentation/trace/hisi-ptt.rst
>
> diff --git a/Documentation/trace/hisi-ptt.rst b/Documentation/trace/hisi-ptt.rst
> new file mode 100644
> index 000000000000..13677705ee1f
> --- /dev/null
> +++ b/Documentation/trace/hisi-ptt.rst
> @@ -0,0 +1,303 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +======================================
> +HiSilicon PCIe Tune and Trace device
> +======================================
> +
> +Introduction
> +============
> +
> +HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
> +integrated Endpoint (RCiEP) device, providing the capability
> +to dynamically monitor and tune the PCIe link's events (tune),
> +and trace the TLP headers (trace). The two functions are independent,
> +but is recommended to use them together to analyze and enhance the
> +PCIe link's performance.
> +
> +On Kunpeng 930 SoC, the PCIe Root Complex is composed of several
> +PCIe cores. Each PCIe core includes several Root Ports and a PTT
> +RCiEP, like below. The PTT device is capable of tuning and
> +tracing the links of the PCIe core.
> +::
> + +--------------Core 0-------+
> + | | [ PTT ] |
> + | | [Root Port]---[Endpoint]
> + | | [Root Port]---[Endpoint]
> + | | [Root Port]---[Endpoint]
> + Root Complex |------Core 1-------+
> + | | [ PTT ] |
> + | | [Root Port]---[ Switch ]---[Endpoint]
> + | | [Root Port]---[Endpoint] `-[Endpoint]
> + | | [Root Port]---[Endpoint]
> + +---------------------------+
> +
> +The PTT device driver registers one PMU device for each PTT device.
> +The name of each PTT device is composed of 'hisi_ptt' prefix with
> +the id of the SICL and the Core where it locates. The Kunpeng 930
> +SoC encapsulates multiple CPU dies (SCCL, Super CPU Cluster) and
> +IO dies (SICL, Super I/O Cluster), where there's one PCIe Root
> +Complex for each SICL.
> +::
> + /sys/devices/hisi_ptt<sicl_id>_<core_id>
> +
> +Tune
> +====
> +
> +PTT tune is designed for monitoring and adjusting PCIe link parameters (events).
> +Currently we support events in 4 classes. The scope of the events
> +covers the PCIe core to which the PTT device belongs.
> +
> +Each event is presented as a file under $(PTT PMU dir)/tune, and
> +a simple open/read/write/close cycle will be used to tune the event.
> +::
> + $ cd /sys/devices/hisi_ptt<sicl_id>_<core_id>/tune
> + $ ls
> + qos_tx_cpl qos_tx_np qos_tx_p
> + tx_path_rx_req_alloc_buf_level
> + tx_path_tx_req_alloc_buf_level
> + $ cat qos_tx_dp
> + 1
> + $ echo 2 > qos_tx_dp
> + $ cat qos_tx_dp
> + 2
> +
> +Current value (numerical value) of the event can be simply read
> +from the file, and the desired value written to the file to tune.
> +
> +1. Tx path QoS control
> +------------------------
> +
> +The following files are provided to tune the QoS of the tx path of
> +the PCIe core.
> +
> +- qos_tx_cpl: weight of Tx completion TLPs
> +- qos_tx_np: weight of Tx non-posted TLPs
> +- qos_tx_p: weight of Tx posted TLPs
> +
> +The weight influences the proportion of certain packets on the PCIe link.
> +For example, for the storage scenario, increase the proportion
> +of the completion packets on the link to enhance the performance as
> +more completions are consumed.
> +
> +The available tune data of these events is [0, 1, 2].
> +Writing a negative value will return an error, and out of range
> +values will be converted to 2. Note that the event value just
> +indicates a probable level, but is not precise.
> +
> +2. Tx path buffer control
> +-------------------------
> +
> +Following files are provided to tune the buffer of tx path of the PCIe core.
> +
> +- tx_path_rx_req_alloc_buf_level: watermark of Rx requested
> +- tx_path_tx_req_alloc_buf_level: watermark of Tx requested
> +
> +These events influence the watermark of the buffer allocated for each
> +type. Rx means the inbound while Tx means outbound. The packets will
> +be stored in the buffer first and then transmitted either when the
> +watermark reached or when timed out. For a busy direction, you should
> +increase the related buffer watermark to avoid frequently posting and
> +thus enhance the performance. In most cases just keep the default value.
> +
> +The available tune data of above events is [0, 1, 2].
> +Writing a negative value will return an error, and out of range
> +values will be converted to 2. Note that the event value just
> +indicates a probable level, but is not precise.
> +
> +Trace
> +=====
> +
> +PTT trace is designed for dumping the TLP headers to the memory, which
> +can be used to analyze the transactions and usage condition of the PCIe
> +Link. You can choose to filter the traced headers by either requester ID,
> +or those downstream of a set of Root Ports on the same core of the PTT
> +device. It's also supported to trace the headers of certain type and of
> +certain direction.
> +
> +You can use the perf command `perf record` to set the parameters, start
> +trace and get the data. It's also supported to decode the trace
> +data with `perf report`. The control parameters for trace is inputted
> +as event code for each events, which will be further illustrated later.
> +An example usage is like
> +::
> + $ perf record -e hisi_ptt0_2/filter=0x80001,type=1,direction=1,
> + format=1/ -- sleep 5
> +
> +This will trace the TLP headers downstream root port 0000:00:10.1 (event
> +code for event 'filter' is 0x80001) with type of posted TLP requests,
> +direction of inbound and traced data format of 8DW.
> +
> +1. filter
> +---------
> +
> +The TLP headers to trace can be filtered by the Root Ports or the requester
> +ID of the endpoints, which are located on the same core of the PTT device.
> +You can set the filter by specifying the `filter` parameter which is required
> +to start the trace. The parameter value is 20 bit. The supported filters and
> +related values are outputted through `available_root_port_filters` and
> +`available_requester_filters` sysfs attributes for Root Ports and Requesters
> +respectively.
> +::
> + $ cat available_root_port_filters
> + 0000:00:10.0 0x80001
> + 0000:00:11.0 0x80004
> + $ cat available_requester_filters
> + 0000:01:00.0 0x00100
> + 0000:01:00.1 0x00101
> +
> +Note that multiple Root Ports can be specified at one time, but only
> +one Endpoint function can be specified in one trace. Specifying both
> +Root Port and function at the same time is not supported.
> +
> +If no filter is available, reading the related filter sysfs attribute
> +will get an empty string.
> +::
> + $ cat available_root_port_filters
> +
> + $ cat available_requester_filters
> +
> +The available filters can be dynamically updated, which means you can always
> +get correct filter information when hotplug events happen, or when you manually
> +remove/rescan the devices.
> +
> +2. type
> +-------
> +
> +You can trace the TLP headers of certain types by specifying the `type`
> +parameter, which is required to start the trace. The parameter value is
> +8 bit. Current supported types and related values are shown below:
> +
> +8'b00000001: posted requests (P)
> +8'b00000010: non-posted requests (NP)
> +8'b00000100: completions (CPL)
> +
> +You can specify multiple types when tracing inbound TLP headers, but can only
> +specify one when tracing outbound TLP headers.
> +
> +3. direction
> +------------
> +
> +You can trace the TLP headers from certain direction, which is relative
> +to the Root Port or the PCIe core, by specifying the `direction` parameter.
> +This is optional and the default parameter is inbound. The parameter value
> +is 4 bit. When the desired format is 4DW, directions and related values
> +supported are shown below:
> +
> +4'b0000: inbound TLPs (P, NP, CPL)
> +4'b0001: outbound TLPs (P, NP, CPL)
> +4'b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B)
> +4'b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A)
> +
> +When the desired format is 8DW, directions and related values supported are
> +shown below:
> +
> +4'b0000: reserved
> +4'b0001: outbound TLPs (P, NP, CPL)
> +4'b0010: inbound TLPs (P, NP, CPL B)
> +4'b0011: inbound TLPs (CPL A)
> +
> +Inbound completions are classified into two types:
> +
> +completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B
> +completion B (CPL B): completion of DMA remote2local and P2P non-posted requests
> +
> +4. format
> +--------------
> +
> +You can change the format of the traced TLP headers by specifying the
> +`format` parameter. The default format is 4DW. The parameter value is 4 bit.
> +Current supported formats and related values are shown below:
> +
> +4'b0000: 4DW length per TLP header
> +4'b0001: 8DW length per TLP header
> +
> +The traced TLP header format is different from the PCIe standard.
> +
> +When using the 8DW data format, the entire TLP header is logged
> +(Header DW0-3 shown below). For example, the TLP header for Memory
> +Reads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17;
> +the header for Configuration Requests is shown in Figure 2.20, etc.
> +
> +In addition, 8DW trace buffer entries contain a timestamp and
> +possibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0).
> +Otherwise this field will be all 0.
> +
> +The bit[31:11] of DW0 is always 0x1fffff, which can be
> +used to distinguish the data format. 8DW format is like
> +::
> + bits [ 31:11 ][ 10:0 ]
> + |---------------------------------------|-------------------|
> + DW0 [ 0x1fffff ][ Reserved (0x7ff) ]
> + DW1 [ Prefix ]
> + DW2 [ Header DW0 ]
> + DW3 [ Header DW1 ]
> + DW4 [ Header DW2 ]
> + DW5 [ Header DW3 ]
> + DW6 [ Reserved (0x0) ]
> + DW7 [ Time ]
> +
> +When using the 4DW data format, DW0 of the trace buffer entry
> +contains selected fields of DW0 of the TLP, together with a
> +timestamp. DW1-DW3 of the trace buffer entry contain DW1-DW3
> +directly from the TLP header.
> +
> +4DW format is like
> +::
> + bits [31:30] [ 29:25 ][24][23][22][21][ 20:11 ][ 10:0 ]
> + |-----|---------|---|---|---|---|-------------|-------------|
> + DW0 [ Fmt ][ Type ][T9][T8][TH][SO][ Length ][ Time ]
> + DW1 [ Header DW1 ]
> + DW2 [ Header DW2 ]
> + DW3 [ Header DW3 ]
> +
> +5. memory management
> +--------------------
> +
> +The traced TLP headers will be written to the memory allocated
> +by the driver. The hardware accepts 4 DMA address with same size,
> +and writes the buffer sequentially like below. If DMA addr 3 is
> +finished and the trace is still on, it will return to addr 0.
> +::
> + +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-+
> + +---------------------------------------------------------+
> +
> +Driver will allocate each DMA buffer of 4MiB. The finished buffer
> +will be copied to the perf AUX buffer allocated by the perf core.
> +Once the AUX buffer is full while the trace is still on, driver
> +will commit the AUX buffer first and then apply for a new one with
> +the same size. The size of AUX buffer is default to 16MiB. User can
> +adjust the size by specifying the `-m` parameter of the perf command.
> +
> +Note that there is a gap between committing the old AUX buffer and
> +applying a new one, which means the trace is stopped during the
> +moment and TLPs transferred in the moment cannot be traced. To avoid
> +this situation, you should begin the trace with large AUX buffer
> +enough to avoid this gap.
> +
> +6. decoding
> +-----------
> +
> +You can decode the traced data with `perf report -D` command (currently
> +only support to dump the raw trace data). The traced data will be decoded
> +according to the format described previously (take 8DW as an example):
> +::
> + [...perf headers and other information]
> + . ... HISI PTT data: size 4194304 bytes
> + . 00000000: 00 00 00 00 Prefix
> + . 00000004: 01 00 00 60 Header DW0
> + . 00000008: 0f 1e 00 01 Header DW1
> + . 0000000c: 04 00 00 00 Header DW2
> + . 00000010: 40 00 81 02 Header DW3
> + . 00000014: 33 c0 04 00 Time
> + . 00000020: 00 00 00 00 Prefix
> + . 00000024: 01 00 00 60 Header DW0
> + . 00000028: 0f 1e 00 01 Header DW1
> + . 0000002c: 04 00 00 00 Header DW2
> + . 00000030: 40 00 81 02 Header DW3
> + . 00000034: 02 00 00 00 Time
> + . 00000040: 00 00 00 00 Prefix
> + . 00000044: 01 00 00 60 Header DW0
> + . 00000048: 0f 1e 00 01 Header DW1
> + . 0000004c: 04 00 00 00 Header DW2
> + . 00000050: 40 00 81 02 Header DW3
> + [...]