Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 4 Apr 2023 15:24:24 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Liang Kan <kan.liang@linux.intel.com>,
        <linux-cxl@vger.kernel.org>, <peterz@infradead.org>
CC:     <mingo@redhat.com>, <acme@kernel.org>, <mark.rutland@arm.com>,
        <will@kernel.org>, <dan.j.williams@intel.com>,
        <linuxarm@huawei.com>, <linux-perf-users@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Dave Jiang <dave.jiang@intel.com>
Subject: RE: [PATCH v4 5/5] docs: perf: Minimal introduction the the CXL PMU
 device and driver
Message-ID: <642ca39865e8b_21a8294d9@dwillia2-xfh.jf.intel.com.notmuch>
References: <20230330164556.31533-1-Jonathan.Cameron@huawei.com>
 <20230330164556.31533-6-Jonathan.Cameron@huawei.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230330164556.31533-6-Jonathan.Cameron@huawei.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?T0fUNhowrTw49VM4z6wOoI6PLT4dHwqNbqCvgGAJxzEnIKPU1Xq3M/lg18V3?=
 =?us-ascii?Q?AD0dkz7oqy2AiwPs3qBMNWYbL0IDVNemEIQCj9G5KdZ7a1ohqzbAF/HE5q3m?=
 =?us-ascii?Q?YFZTesFyrTsc8guHxoisn7upJaRb43rcTUOHIbWwMHfzxLSyvi8Uou/wGt2J?=
 =?us-ascii?Q?/JAWmWc75UKitTihxRK/f4FhJw+5cF7sBCCqpY9gJBH+m2C4NSIeF/yza0w0?=
 =?us-ascii?Q?/c+4qjxlh9eNiGEHk1lWrFltC+Wm4PhFmRHvjToGBnQSJ+dTDrWOWtANEVao?=
 =?us-ascii?Q?VTfguMu+SP89aZh0tKxC2yS9UH7Fhaez6rAUS9stgs7ZGHXPm3SW/tYz9aQe?=
 =?us-ascii?Q?9AhKlpt7aWZmHptrXaaV9oUNnlpc5KEpTXsDP6ITispEuYkM592UnfkBUzoV?=
 =?us-ascii?Q?I+PZTyuY+r9GpEeEQ2jF/TKCtGZuCpW3oXGKo1eVnoErV3Elg4Uq32PnvsTj?=
 =?us-ascii?Q?D0gd+vZlRWkJ1jZb8q9WT3GWEBaXQWxMDbnDB/j9gEkXRcTJu5lZEU6VqZHp?=
 =?us-ascii?Q?3D79IxiNiDx/639LvkgtekQStR+HmCjAwgSoitZrZEqBdZlIpehLUob/WYH0?=
 =?us-ascii?Q?2YTtjcttotSLsO/0alJrwizdrKjtjDgNDgkr7WrgOrwQ4ezFWhR28NtcelH8?=
 =?us-ascii?Q?rO2e34Oim/bud+n9f7NP0iJsheuGtgu3DGdSaSzSezysde92tEvlawPbsm4O?=
 =?us-ascii?Q?b5S8ijD9+DSnGLMgxgFu8o0oufXGATNBIuJPijjDbjw5jsJbwXxhApw1PwSp?=
 =?us-ascii?Q?6uiUWar8HOrXbUI14kc5UA5iiRzuO7JGcL9YOzFKhBB5n9ErftVSffprzga3?=
 =?us-ascii?Q?f6AZ7CY239bjitM3oHQvpnH1MJXZB87o3JFA8hdyTucVCHCCLUs05ocT87bB?=
 =?us-ascii?Q?Yhc+Ehfe8HMAG3KRDOmzhnyOqKdRARDu91pf3dwVukmW6ZJZWTxBnFJ9dKPZ?=
 =?us-ascii?Q?w0Ocl+/Fuk7etpQ1w2yrOfDVhigF5pT6FE5IKjFiMZPssTU1DnAmR8ibugkh?=
 =?us-ascii?Q?yi6AxwPjtSfz9oyRu4mXeexQDAIDbL+dkQgNSchZIR4ZhD0NoUef9dPTjHe+?=
 =?us-ascii?Q?FuQKN3vnNOWWqUT6J1uWu5stwDBJfBHKb1GVjav3tqqdgV5lpCAfZUMs/rY9?=
 =?us-ascii?Q?tUp5pHymo4YRHA3P5ys+qdCcFE1sttsPwyk1wy8a9NnUM7X5FJYO3n8cZWyt?=
 =?us-ascii?Q?Aq/puCQmxkFA4ruS+0/f+xnCct/Zdf+B0pbDNbw3KuqctPNkBfct29+zgRAg?=
 =?us-ascii?Q?Ma0a55dAhWReMiDb5MaNd6wIau6CTnRblelAVDFE9grRIiTbMhlRGy1+oMOr?=
 =?us-ascii?Q?q7CLa1+G9ri7KpT3JVIl6PrXoMAzq4iOjFhz4Dm+yPKqV6Me8JJ8Bg0xPjlo?=
 =?us-ascii?Q?UQAqNyuW24H/ns4HfrWE8n4vvxsF5xpW8C/2KaYDCJLeiyHt1Bk4sP22X79E?=
 =?us-ascii?Q?opp0f/D8fN6kkU7WopuZxsiJjQ/lGj76rzinkDJcuu9YdQGSeR/TyZjd35Sw?=
 =?us-ascii?Q?GHvXyMuaBU2A5Y+Gxwqdr1uuQ2oBoij1wCw2lqvPGsZvWej9UwaWmSWSRnfL?=
 =?us-ascii?Q?bFfxLj01dfjVEmjIpxxE55eyBMcAyAJSTWn7h7PkBZMNHJk0UxL+IA7GNrpR?=
 =?us-ascii?Q?5w=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a5b3629f-1a7b-4584-620d-08db355b5a2a
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Apr 2023 22:24:27.6300
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 6xu2uVSxNb3xuLvK1BBP4PFedjmWrmBcF4UO/cUzwflNYcwZ/92hEcT5hbYn9Jeqg1BuVir0lwUTHd8AhJ3txi1LBZq45v1Gk09ICp9iy7k=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR11MB4978
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE,
        SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Jonathan Cameron wrote:
> Very basic introduction to the device and the current driver support
> provided. I expect to expand on this in future versions of this patch
> set.
> 
> Reviewed-by: Dave Jiang <dave.jiang@intel.com>
> Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> 
> --
> v4: No change
> ---
>  Documentation/admin-guide/perf/cxl.rst   | 65 ++++++++++++++++++++++++
>  Documentation/admin-guide/perf/index.rst |  1 +
>  2 files changed, 66 insertions(+)
> 
> diff --git a/Documentation/admin-guide/perf/cxl.rst b/Documentation/admin-guide/perf/cxl.rst
> new file mode 100644
> index 000000000000..46235dff4b21
> --- /dev/null
> +++ b/Documentation/admin-guide/perf/cxl.rst
> @@ -0,0 +1,65 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +======================================
> +CXL Performance Monitoring Unit (CPMU)
> +======================================
> +
> +The CXL rev 3.0 specification provides a definition of CXL Performance
> +Monitoring Unit in section 13.2: Performance Monitoring.
> +
> +CXL components (e.g. Root Port, Switch Upstream Port, End Point) may have
> +any number of CPMU instances. CPMU capabilities are fully discoverable from
> +the devices. The specification provides event definitions for all CXL protocol
> +message types and a set of additional events for things commonly counted on
> +CXL devices (e.g. DRAM events).
> +
> +CPMU driver
> +===========
> +
> +The CPMU driver register a perf PMU with the name cpmu<id> on the CXL bus.

s/register/registers/

> +
> +    /sys/bus/cxl/device/cpmu<id>
> +
> +The associated PMU is registered as
> +
> +   /sys/bus/event_sources/devices/cpmu<id>
> +
> +In common with other CXL bus devices, the id has no specific meaning and the
> +relationship to specific CXL device should be established via the device parent
> +of the device on the CXL bus.

So I went to go add some text about how to identify PMUs in a persistent
manner from one boot to the next. For CXL memdevs this is done by the
'serial' attribute which is always stable regardless of the device init
order. That's harder to get to from the pmu device because it may be
associated with a device that does not have a memdev.

I think it's also going to be frustrating for userspace to see
randomized pmu ids across devices since that probing will happen in
parallel. So how about:

1/ Add serial as an attribute for each PMU to export
2/ Change the device name format to be "pmuX.Y" where X can just reuse
the memdev id for endpoints and be another value for switches, and Y is
guaranteed to be 0-based and in hardware discovery order.

...with that, someone can write a udev script that can persistently
identify PMU[Y] on device[serial] each boot.

That also cleans up a /sys/bus/cxl/devices listing to make it clear
which pmu instances belong together.
 
> +
> +PMU driver provides description of available events and filter options in sysfs.
> +
> +The "format" directory describes all formats of the config (event vendor id,
> +group id and mask) config1 (threshold, filter enables) and config2 (filter
> +parameters) fields of the perf_event_attr structure.  The "events" directory
> +describes all documented events show in perf list.
> +
> +The events shown in perf list are the most fine grained events with a single
> +bit of the event mask set. More general events may be enable by setting
> +multiple mask bits in config. For example, all Device to Host Read Requests
> +may be captured on a single counter by setting the bits for all of
> +
> +* d2h_req_rdcurr
> +* d2h_req_rdown
> +* d2h_req_rdshared
> +* d2h_req_rdany
> +* d2h_req_rdownnodata
> +
> +Example of usage::
> +
> +  $#perf list
> +  cpmu0/clock_ticks/                                 [Kernel PMU event]
> +  cpmu0/d2h_req_itomwr/                              [Kernel PMU event]
> +  cpmu0/d2h_req_rdany/                               [Kernel PMU event]
> +  cpmu0/d2h_req_rdcurr/                              [Kernel PMU event]
> +  -----------------------------------------------------------
> +
> +  $# perf stat -e cpmu0/clock_ticks/ -e cpmu0/d2h_req_itowrm/

Ah here's the examples I was looking for in the last patch, nice.

> +
> +Vendor specific events may also be available and if so can be used via
> +
> +  $# perf stat -e cpmu0/vid=VID,gid=GID,mask=MASK/
> +
> +The driver does not support sampling. So "perf record" and attaching to
> +a task are unsupported.

Is this a common restriction for CPU-external pmus, or do you see
sampling support required to get this upstream?