Date:   Mon, 6 Mar 2023 10:27:22 +0000
From:   Jonathan Cameron <Jonathan.Cameron@huawei.com>
To:     Dave Jiang <dave.jiang@intel.com>
CC:     <linux-cxl@vger.kernel.org>, <peterz@infradead.org>,
        <mingo@redhat.com>, <acme@kernel.org>, <mark.rutland@arm.com>,
        <will@kernel.org>, <dan.j.williams@intel.com>,
        <bwidawsk@kernel.org>, <ira.weiny@intel.com>,
        <vishal.l.verma@intel.com>, <alison.schofield@intel.com>,
        <linuxarm@huawei.com>, <linux-perf-users@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 4/4] docs: perf: Minimal introduction the the CXL PMU
 device and driver.
Message-ID: <20230306102722.00005ead@huawei.com>
In-Reply-To: <4dabd301-32d7-968e-e048-8135b072a7b5@intel.com>
References: <20230303175022.10806-1-Jonathan.Cameron@huawei.com>
        <20230303175022.10806-5-Jonathan.Cameron@huawei.com>
        <4dabd301-32d7-968e-e048-8135b072a7b5@intel.com>
Organization: Huawei Technologies R&D (UK) Ltd.
MIME-Version: 1.0
Content-Type: text/plain; charset="US-ASCII"
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Fri, 3 Mar 2023 11:34:37 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> On 3/3/23 10:50 AM, Jonathan Cameron wrote:
> > Very basic introduction to the device and the current driver support
> > provided. I expect to expand on this in future versions of this patch
> > set.
> > 
> > Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>  
> 
> Reviewed-by: Dave Jiang <dave.jiang@intel.com>
> 
> > 
> > --
> > v1:
> > - Add docs for how to use a Vendor Defined Counter.
> > RFC:
> > - I'll post separately about this shortly, but it seems very odd
> >    to me that there is no way to assign a parent to an event_sources
> >    device.  As a result we get the messy approach of playing match
> >    the name to figure out what the CPMU instance is connected to.  
> 
> Would it be too awkward to encode the parent name into the cpmu name?

We could, though it gets messy fast as we have multiple instances per
CXL component - right now I'm only registering them for the PCI EPs but
we also need to deal with all the other places they could be.
We can also expose extra information via additional attributes, but where
I've seen that done in the past it's a topology description (i.e. socket X,
die Y) and our topology isn't stable.

Even if we do I'd like to fix the issue with parents for event_sources
and once that's done I'm not sure we need to worry so much about how
to do the reverse lookup.

Input from perf / event_sources folk on how they would like to do
the association needed!

Jonathan


> 
> > ---
> >   Documentation/admin-guide/perf/cxl.rst   | 65 ++++++++++++++++++++++++
> >   Documentation/admin-guide/perf/index.rst |  1 +
> >   2 files changed, 66 insertions(+)
> > 
> > diff --git a/Documentation/admin-guide/perf/cxl.rst b/Documentation/admin-guide/perf/cxl.rst
> > new file mode 100644
> > index 000000000000..46235dff4b21
> > --- /dev/null
> > +++ b/Documentation/admin-guide/perf/cxl.rst
> > @@ -0,0 +1,65 @@
> > +.. SPDX-License-Identifier: GPL-2.0
> > +
> > +======================================
> > +CXL Performance Monitoring Unit (CPMU)
> > +======================================
> > +
> > +The CXL rev 3.0 specification provides a definition of CXL Performance
> > +Monitoring Unit in section 13.2: Performance Monitoring.
> > +
> > +CXL components (e.g. Root Port, Switch Upstream Port, End Point) may have
> > +any number of CPMU instances. CPMU capabilities are fully discoverable from
> > +the devices. The specification provides event definitions for all CXL protocol
> > +message types and a set of additional events for things commonly counted on
> > +CXL devices (e.g. DRAM events).
> > +
> > +CPMU driver
> > +===========
> > +
> > +The CPMU driver register a perf PMU with the name cpmu<id> on the CXL bus.
> > +
> > +    /sys/bus/cxl/device/cpmu<id>
> > +
> > +The associated PMU is registered as
> > +
> > +   /sys/bus/event_sources/devices/cpmu<id>
> > +
> > +In common with other CXL bus devices, the id has no specific meaning and the
> > +relationship to specific CXL device should be established via the device parent
> > +of the device on the CXL bus.
> > +
> > +PMU driver provides description of available events and filter options in sysfs.
> > +
> > +The "format" directory describes all formats of the config (event vendor id,
> > +group id and mask) config1 (threshold, filter enables) and config2 (filter
> > +parameters) fields of the perf_event_attr structure.  The "events" directory
> > +describes all documented events show in perf list.
> > +
> > +The events shown in perf list are the most fine grained events with a single
> > +bit of the event mask set. More general events may be enable by setting
> > +multiple mask bits in config. For example, all Device to Host Read Requests
> > +may be captured on a single counter by setting the bits for all of
> > +
> > +* d2h_req_rdcurr
> > +* d2h_req_rdown
> > +* d2h_req_rdshared
> > +* d2h_req_rdany
> > +* d2h_req_rdownnodata
> > +
> > +Example of usage::
> > +
> > +  $#perf list
> > +  cpmu0/clock_ticks/                                 [Kernel PMU event]
> > +  cpmu0/d2h_req_itomwr/                              [Kernel PMU event]
> > +  cpmu0/d2h_req_rdany/                               [Kernel PMU event]
> > +  cpmu0/d2h_req_rdcurr/                              [Kernel PMU event]
> > +  -----------------------------------------------------------
> > +
> > +  $# perf stat -e cpmu0/clock_ticks/ -e cpmu0/d2h_req_itowrm/
> > +
> > +Vendor specific events may also be available and if so can be used via
> > +
> > +  $# perf stat -e cpmu0/vid=VID,gid=GID,mask=MASK/
> > +
> > +The driver does not support sampling. So "perf record" and attaching to
> > +a task are unsupported.
> > diff --git a/Documentation/admin-guide/perf/index.rst b/Documentation/admin-guide/perf/index.rst
> > index 9de64a40adab..f60be04e4e33 100644
> > --- a/Documentation/admin-guide/perf/index.rst
> > +++ b/Documentation/admin-guide/perf/index.rst
> > @@ -21,3 +21,4 @@ Performance monitor support
> >      alibaba_pmu
> >      nvidia-pmu
> >      meson-ddr-pmu
> > +   cxl