Received-SPF: pass (google.com: domain of linux-kernel+bounces-214193-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <e649a49f-344f-4dbd-be2f-d70f932a80f4@intel.com>
Date: Thu, 13 Jun 2024 18:48:28 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v4 14/19] x86/resctrl: Add the interface to assign ABMC
 counter
To: Babu Moger <babu.moger@amd.com>, <corbet@lwn.net>, <fenghua.yu@intel.com>,
	<tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>,
	<dave.hansen@linux.intel.com>
CC: <x86@kernel.org>, <hpa@zytor.com>, <paulmck@kernel.org>,
	<rdunlap@infradead.org>, <tj@kernel.org>, <peterz@infradead.org>,
	<yanjiewtw@gmail.com>, <kim.phillips@amd.com>, <lukas.bulwahn@gmail.com>,
	<seanjc@google.com>, <jmattson@google.com>, <leitao@debian.org>,
	<jpoimboe@kernel.org>, <rick.p.edgecombe@intel.com>,
	<kirill.shutemov@linux.intel.com>, <jithu.joseph@intel.com>,
	<kai.huang@intel.com>, <kan.liang@linux.intel.com>,
	<daniel.sneddon@linux.intel.com>, <pbonzini@redhat.com>,
	<sandipan.das@amd.com>, <ilpo.jarvinen@linux.intel.com>,
	<peternewman@google.com>, <maciej.wieczor-retman@intel.com>,
	<linux-doc@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
	<eranian@google.com>, <james.morse@arm.com>
References: <cover.1716552602.git.babu.moger@amd.com>
 <631092558e7fe0ac2e6267070e40c4a97b300f57.1716552602.git.babu.moger@amd.com>
From: Reinette Chatre <reinette.chatre@intel.com>
Content-Language: en-US
In-Reply-To: <631092558e7fe0ac2e6267070e40c4a97b300f57.1716552602.git.babu.moger@amd.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?K3U5b0lSQUZJandtL2hMV21yRFJ5VjZJTGZrck0rMS9zSC9vMnA3Y1dWVnRn?=
 =?utf-8?B?UldLYjZiQ1JPUjFyZC9vem9KMUQrRHMwOUJ0ZzE1K0V5MEZ2R1Z0Qk9lQTVY?=
 =?utf-8?B?Q1BLTktweTZQVU95Zm5VaFFQZVkwQ0x1R0lFUmdqc0ZPKy80VzErc05uQ2pY?=
 =?utf-8?B?b1B3QmF5c1ZBcnB0WExhaFd0dHFnaHFTRGlQOW8vVlUxTWRzSXJVZEk4NWxL?=
 =?utf-8?B?bVhUSWJ2N1ZhUk1wNHcvSHZpdlpuWVoyam9nOGtpZVJNWVJrc3BuaW9oRWFi?=
 =?utf-8?B?M3pPdVdqbFl3RkdZUzNTbVVxTldYb2lLVDFOc0FRTzB6Ui92aFlJbmVMdTkv?=
 =?utf-8?B?MXVnRStVbXVCTGtzdWJUWU5yRllHNmMralU3NEdsT1IxMGhwdnE1NlllcEtG?=
 =?utf-8?B?UmoyRHo3NnhzeUh1VWtLQ2poeFVYVzh2Sm9aVWp2MUhJR2hNbC9QaVJ5Y1Vr?=
 =?utf-8?B?a3hhaGNiWTY2OEkzVG1tRG1lTldGTG9BU1hLa29RdVhoOGJobjlRM1JLWnRL?=
 =?utf-8?B?Z0FFR2U0UVRoVENuTkZlZHNDS244U1Byd1FuR3J2ZzdacUx6V2F4bzhRczd1?=
 =?utf-8?B?ZHhGMGVZQUpzaEIrcGx0akEwRlNpN2F2bnh3SHcvMVRLMEtROVIwLzQ1NzdZ?=
 =?utf-8?B?djBlcmhxOUhVRnZQTDZuSmVrQ2htc3dZS3BGRm1IZitPelJZR3p2bzk0MkZk?=
 =?utf-8?B?L1RGOVRVRUdVZWtiekVUcHpROWprQWlmUzdTTVhCR3V1VnhLaFkwbkJrZjZC?=
 =?utf-8?B?RjN1T2YvcHMwbmF6SlNoTmZ1Wm8yWDY3Vk5vaFBlUGlvTGJaU0tHeWtjUGhZ?=
 =?utf-8?B?U1k5VElNaC9IOWR6dHpkbVc5ZlVWVXRNUk5NOXhxT0xWaEJ6QjFQeWNLa0FP?=
 =?utf-8?B?VVBWQmZOYVc3dk5TWXkzeGQ0NXY0LzVJQVpQYkhxbWt4VFU5NzRhaXptVzJX?=
 =?utf-8?B?UWJibVNqOU5aOGQram9ML09GZnU4ZkMxOU00S3dBWnRTU0hNTkR5YmJ3RVlu?=
 =?utf-8?B?Sk5YRVlLVmFyZWxERHRGemR0RmoxbStNVSs2K2tLNEw3RGc4RnQyQWdqUFBs?=
 =?utf-8?B?VHIzUUtLclhEQVV0bmxOdUtENy9SODdzOVB2eGpOVmZ6SFJyeEx2aFEwRyty?=
 =?utf-8?B?T0thS1NXeVA0V0MwN2tURFV4M1doZG1ZUWRJZmM1bFBqQVgxdkxOL1FzRFJm?=
 =?utf-8?B?bGd1cHcyd3M2Sjd4R0tOUE9JVUtOTG9CbGw2QkVmMThXSFF1eWJ3K1BoZkVZ?=
 =?utf-8?B?QWxhZG5VTzBQQ1htNWhGUEQ5aFMyek5wR2xqRlpERUh1WnJ1MDdtb2diR1Rz?=
 =?utf-8?B?YjhOanZVQ2JWMGdsMFZ1eVlGTkVRdmp2NWxxSjhaa2NuZ0d2SjljWVl6b05S?=
 =?utf-8?B?K2NPdms1cE96ZWFWT3ZPTWRQVzVRaFRUTUNQcEZmMXNtM3pEK1BuZGlIRjNQ?=
 =?utf-8?B?aFlXb0JDVnV2V2NablV2TGRWRGNYeDNwbXVvWUpQMUEzQVdUa0gyRTlpN2lB?=
 =?utf-8?B?NGVGTkJ4TzYzdDR3dzBSMmxkTDFQUnlYOVBObTk0ajN0OGkyUGgyTlQrVm9W?=
 =?utf-8?B?YTFRNWtDWGg1Zk1HZDVlWHdPbUZTWFdNZjFRblQ1YWt1TmFVVE1lZnFIRVlu?=
 =?utf-8?B?YTYrQlZYOFNaT2lyUDhDSStnSjR5bEo3Wng2ZEgyZFYzeGV4SUNxdzIvNGlG?=
 =?utf-8?B?NXZWWU1QUHM3ck9xcE82V04rcUcrenA4RHZFMTVaRlpMR2RMZEFpT3FTeHZD?=
 =?utf-8?B?QWJscnpyU0tZTmplaWFJRkJDQ2hWUGx5NWVWclhrRHVZR2NZNU5SbXA5TWdK?=
 =?utf-8?B?d3R5UFFtdzAvSTVFSmM4dmtCZjFHUU9maFlBVWFoNjZBMTFNbTc1NVhQZGpM?=
 =?utf-8?B?cHNUeDgxN3ZHTGd4V3AxM2pVWGFNTDBsaWhqTDRmakZ4clVzNEt5cDlPUU9D?=
 =?utf-8?B?Q1NOMjBZZ21wKzhDbnJlT2lhdWdLNVZSNUlqUzdYOVBreWVzeFZQQ0xaaDdF?=
 =?utf-8?B?QXVtKzFST0RRNnF2cGdSZnB5VTcvbHpQMUFZMkoyNzRMSncwSVBjMWdaZGZ4?=
 =?utf-8?B?SlRXTkg5dmVIaGY2VC9yS1hmeWdiNkRBZ2xoT3YzZVNQb2ZxeDVxL3FMNk9I?=
 =?utf-8?B?RXlhMGNCWFpyUmZMbUZYdTByYU9WMGV1MzhIV3dBLzNRczdwREhPZUhsbHd6?=
 =?utf-8?B?L3c9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: fcf9d3bd-9878-4d85-4b11-08dc8c1424ed
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jun 2024 01:48:52.7600
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: RrhgO5cR5yCNxeunKjpz7GLl2dmX2Zr4XdAZRmmyaayKA6EjiVbQ0u8Z+yYokkq42mLltnGOEOnVZ/QcWoH+lD6QN+gVTrFqbxnYb3x5B44=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR11MB6203
X-OriginatorOrg: intel.com

Hi Babu,

On 5/24/24 5:23 AM, Babu Moger wrote:
> ABMC feature requires to users to assign a hardware counter to an RMID

"requires to users" -> "requires users"?

> to monitor the events. Provide the interfaces to assign a counter.
> 
> Individual counters are configured by writing to L3_QOS_ABMC_CFG MSR
> and specifying the counter id, bandwidth source, and bandwidth types.

Where is discription of what this patch does and why?

This and following patches seem to introduce building blocks that
are only used later in series. These building blocks are introduced
without any information about what they do and why and that makes it
difficult to understand how this work will fall into place.

> 
> The feature details are documented in the APM listed below [1].
> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
>      Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth
>      Monitoring (ABMC).
> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> ---
> v4: Commit message update.
>      User bitmap APIs where applicable.
>      Changed the interfaces considering MPAM(arm).
>      Added domain specific assignment.
> 
> v3: Removed the static from the prototype of rdtgroup_assign_abmc.
>      The function is not called directly from user anymore. These
>      changes are related to global assignment interface.
> 
> v2: Minor text changes in commit message.
> ---
>   arch/x86/kernel/cpu/resctrl/internal.h |   3 +
>   arch/x86/kernel/cpu/resctrl/rdtgroup.c | 101 +++++++++++++++++++++++++
>   2 files changed, 104 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
> index 45ed33f4f0ff..a88c8fc5e4df 100644
> --- a/arch/x86/kernel/cpu/resctrl/internal.h
> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
> @@ -657,6 +657,9 @@ void rdt_domain_reconfigure_cdp(struct rdt_resource *r);
>   void __init resctrl_file_fflags_init(const char *config,
>   				     unsigned long fflags);
>   void arch_domain_mbm_evt_config(struct rdt_hw_domain *hw_dom);
> +int resctrl_arch_assign(struct rdt_domain *d, u32 evtid, u32 rmid,
> +			u32 ctr_id, u32 closid, bool enable);
> +int resctrl_grp_assign(struct rdtgroup *rdtgrp, u32 evtid);
>   void rdt_staged_configs_clear(void);
>   bool closid_allocated(unsigned int closid);
>   int resctrl_find_cleanest_closid(void);
> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> index 0e425c91fa46..48df76499a04 100644
> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> @@ -203,6 +203,19 @@ static void num_cntrs_init(void)
>   	num_cntrs_free_map_len = r->mon.num_cntrs;
>   }
>   
> +static int assign_cntrs_alloc(void)

Only one counter is allocated and "assign" is unexpected since it
seems to imply how this counter will be used.
It can just be "mon_cntr_alloc()"?

> +{
> +	u32 ctr_id = find_first_bit(&num_cntrs_free_map,
> +				    num_cntrs_free_map_len);
> +
> +	if (ctr_id >= num_cntrs_free_map_len)
> +		return -ENOSPC;
> +
> +	__clear_bit(ctr_id, &num_cntrs_free_map);
> +
> +	return ctr_id;
> +}
> +
>   /**
>    * rdtgroup_mode_by_closid - Return mode of resource group with closid
>    * @closid: closid if the resource group
> @@ -1830,6 +1843,94 @@ static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of,
>   	return ret ?: nbytes;
>   }
>   
> +static void rdtgroup_abmc_cfg(void *info)
> +{
> +	u64 *msrval = info;
> +
> +	wrmsrl(MSR_IA32_L3_QOS_ABMC_CFG, *msrval);
> +}
> +

Please add comments to these functions to summarize what
they do.

> +int resctrl_arch_assign(struct rdt_domain *d, u32 evtid, u32 rmid,
> +			u32 ctr_id, u32 closid, bool enable)
> +{
> +	struct rdt_hw_domain *hw_dom = resctrl_to_arch_dom(d);
> +	union l3_qos_abmc_cfg abmc_cfg = { 0 };
> +	struct arch_mbm_state *arch_mbm;
> +
> +	abmc_cfg.split.cfg_en = 1;
> +	abmc_cfg.split.ctr_en = enable ? 1 : 0;
> +	abmc_cfg.split.ctr_id = ctr_id;
> +	abmc_cfg.split.bw_src = rmid;
> +
> +	/*
> +	 * Read the event configuration from the domain and pass it as
> +	 * bw_type.
> +	 */
> +	if (evtid == QOS_L3_MBM_TOTAL_EVENT_ID) {
> +		abmc_cfg.split.bw_type = hw_dom->mbm_total_cfg;
> +		arch_mbm = &hw_dom->arch_mbm_total[rmid];
> +	} else {
> +		abmc_cfg.split.bw_type = hw_dom->mbm_local_cfg;
> +		arch_mbm = &hw_dom->arch_mbm_local[rmid];
> +	}
> +
> +	smp_call_function_any(&d->cpu_mask, rdtgroup_abmc_cfg, &abmc_cfg, 1);
> +
> +	/* Reset the internal counters */

"internal counters"? This needs a definition ... but since this is not
a new data structure the comment can be more specific about what is done
and why.

> +	if (arch_mbm)
> +		memset(arch_mbm, 0, sizeof(struct arch_mbm_state));
> +
> +	return 0;

If this function always succeeds then it can just be void?

> +}
> +
> +int resctrl_grp_assign(struct rdtgroup *rdtgrp, u32 evtid)

Please use consistent naming. Note how the filename is "rdtgroup.c" and this
function operates on "struct rdtgroup" and if you take a closer look at
the functions in this file and the header where you add this function you
will notice a distinct pattern of "rdtgroup_" used as prefix. There really
is no reason to start a new namespace for this.

> +{
> +	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
> +	int ctr_id = 0, index;
> +	struct rdt_domain *d;
> +	u32 mon_state;
> +
> +	index = mon_event_config_index_get(evtid);
> +	if (index == INVALID_CONFIG_INDEX) {
> +		rdt_last_cmd_puts("Invalid event id\n");
> +		return -EINVAL;
> +	}
> +
> +	if (evtid == QOS_L3_MBM_TOTAL_EVENT_ID) {
> +		mon_state = ASSIGN_TOTAL;
> +	} else if (evtid == QOS_L3_MBM_LOCAL_EVENT_ID) {
> +		mon_state = ASSIGN_LOCAL;
> +	} else {
> +		rdt_last_cmd_puts("Invalid event id\n");
> +		return -EINVAL;
> +	}
> +
> +	/* Nothing to do if event has been assigned already */
> +	if (rdtgrp->mon.mon_state & mon_state) {

Why is this mon_state needed? Is it not redundant when considering that
struct mongroup has an array of counter IDs that can already be used
to see if a counter is assigned or not?
What if there is a new:
#define MON_CNTR_UNSET U32_MAX

When cntr_id[index] == MON_CNTR_UNSET then it is not assigned.

> +		rdt_last_cmd_puts("ABMC counter is assigned already\n");
> +		return 0;
> +	}
> +
> +	/*
> +	 * Allocate a new counter and update domains
> +	 */
> +	ctr_id = assign_cntrs_alloc();
> +	if (ctr_id < 0) {
> +		rdt_last_cmd_puts("Out of ABMC counters\n");
> +		return -ENOSPC;
> +	}
> +
> +	rdtgrp->mon.ctr_id[index] = ctr_id;
> +
> +	list_for_each_entry(d, &r->domains, list)
> +		resctrl_arch_assign(d, evtid, rdtgrp->mon.rmid, ctr_id,
> +				    rdtgrp->closid, 1);
> +
> +	rdtgrp->mon.mon_state |= mon_state;
> +
> +	return 0;
> +}
> +
>   /* rdtgroup information files for one cache resource. */
>   static struct rftype res_common_files[] = {
>   	{

Reinette