Received-SPF: pass (google.com: domain of linux-kernel+bounces-168414-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <3149b62a-b9a4-49c6-a757-5a0e36df62df@intel.com>
Date: Fri, 3 May 2024 16:32:04 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH v3 10/17] x86/resctrl: Add data structures for ABMC
 assignment
To: Babu Moger <babu.moger@amd.com>, <corbet@lwn.net>, <fenghua.yu@intel.com>,
	<tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>,
	<dave.hansen@linux.intel.com>
CC: <x86@kernel.org>, <hpa@zytor.com>, <paulmck@kernel.org>,
	<rdunlap@infradead.org>, <tj@kernel.org>, <peterz@infradead.org>,
	<yanjiewtw@gmail.com>, <kim.phillips@amd.com>, <lukas.bulwahn@gmail.com>,
	<seanjc@google.com>, <jmattson@google.com>, <leitao@debian.org>,
	<jpoimboe@kernel.org>, <rick.p.edgecombe@intel.com>,
	<kirill.shutemov@linux.intel.com>, <jithu.joseph@intel.com>,
	<kai.huang@intel.com>, <kan.liang@linux.intel.com>,
	<daniel.sneddon@linux.intel.com>, <pbonzini@redhat.com>,
	<sandipan.das@amd.com>, <ilpo.jarvinen@linux.intel.com>,
	<peternewman@google.com>, <maciej.wieczor-retman@intel.com>,
	<linux-doc@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
	<eranian@google.com>, <james.morse@arm.com>
References: <cover.1711674410.git.babu.moger@amd.com>
 <c3103dd30bc07070380a01922f66208759afe837.1711674410.git.babu.moger@amd.com>
Content-Language: en-US
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <c3103dd30bc07070380a01922f66208759afe837.1711674410.git.babu.moger@amd.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cnUwUVdIeGt0WkNvK1FvaE1EVkwrYWVHdUdYT3FTNDlaRHkrZENqaiszenQ3?=
 =?utf-8?B?dUN0V2JyY1RYSjZ5VlZWOHNFcWljNUE5TGlja0JXV3FiYVc4aTVuSm02b0Vi?=
 =?utf-8?B?cUptNzJyU0hVNzdXeFdya3Y3WkVqQkJpYXRQU2NGd2w4Yk9pVWx3YjB1WVBP?=
 =?utf-8?B?MUlhaEFQaWYwQnBqenZGOUoybXN1YzNkN2JqQ2g3OUxUVk5rMERnZ002SzRq?=
 =?utf-8?B?dVNYdDdoSzUwVzQ3Y0czWHZtbWk0TUROK3YzQllhY2NMMGhpcDFJdHlLSXFQ?=
 =?utf-8?B?UjAvNzEzVUVjbWhXUEUwTWxmQlBncUkrS1M2V3lWd0ZJRFh5R0tYUzdXY3lY?=
 =?utf-8?B?cTJyTW90ekNBNU5qQ1BVeGRrc1picGF1NUo0ekxPcU1TcDMxamJ1YituTWdt?=
 =?utf-8?B?aUFOOXhBdzZlYjdqT29FNlZkRmNXYTJQUllBc0hXRlhjTVF6b2REczZWbXpP?=
 =?utf-8?B?a3A0OTBrelVMTE5sMVV2VnZCNk5sL0grbnJaRncwSStURnk4Lzd6bGxvSVlD?=
 =?utf-8?B?VWJ0VVY4aGZVL2VVWFcrVC93K3M0UXJDVWpNdFF0eDhjOFllR1U2YkhZcHR4?=
 =?utf-8?B?Q1lXalo3cGhiYitNeXNlRTRSK1N3ZWF3K1JyeWJDb2JyenNqcDE2blNzeDhh?=
 =?utf-8?B?NTFZWTFaWlY1ajUxcDdJODVMT1dKVVkrT2pHdXkzdHlKYnZtTUV2U0ZYZkpp?=
 =?utf-8?B?c2djbDZ0RWlUV3c3NFJSRiszTTFsQ0NkL3BGdUxYelFhZDhsOTRnS3oxeEVO?=
 =?utf-8?B?cmFSdUZHRHl5MVU5Q3V3QXU5MEJ3OGFSdUNET2dRV21odTA1WEwwckdDcHk0?=
 =?utf-8?B?cDNoM3dMUDd3TW9qa2FtZm13WUo5MlZNVU4wZ3ZPVk9LL1Rib0gxSmlHOStU?=
 =?utf-8?B?NG04czM1YnlXWjgyU0s2UmVuNnJhSjNUQTVrOUljSjQ0dHJRRnhuNGJjdHNr?=
 =?utf-8?B?ZDRqSEFXQlJrTHl0WG15U2M1VEFTOEVMQjBjTDM1aE5aZXhGZXpBaXVLVFJ3?=
 =?utf-8?B?Nnh6MHR6bU51SGtCR0RGMG1obHFvM1doU1JYTnVJbUx0NEZzWUNBdFhvR3lK?=
 =?utf-8?B?L0lyQUJJNVJLVFhJT1BjRFYvMWEwU3J3NWY3UzhYckk4bmJLUVNYVFlaSzg1?=
 =?utf-8?B?MzRVdXJYdWJJSW1JUUdCK1ZnYWlTZzVxR3ZKaHMzeU9NSDNOelNjNE4rRU9O?=
 =?utf-8?B?MXJLaDFLMHBNMXlIRWVLT2l2YXMzL1VmeVZ4U0JGSXg3Ulg0MDN3bXp0OVZt?=
 =?utf-8?B?UXU2TjlXMjcwa0JQTHhVb3RyYUZoN1ppektVdEp5MzI1cEptY0tHMWVnT3F0?=
 =?utf-8?B?SGQxRXRRa0NzUWt4ajRkN1dWRW1VNVpHTm1oMEJPOGJpK2hnMlZ3SzdseUJj?=
 =?utf-8?B?OXFRYUFQMzVMOXNMR1BoMTkySXVGVXV4bU9ickZUTFhZcFhmUW45anhlSHdW?=
 =?utf-8?B?U1NtTEZDY2NrcURnUnMvbm5FSDEzNXlOVWt0bU5MZ0VUaFF2UTBmREVsY05K?=
 =?utf-8?B?ZVpFeXJrY0FtMGgzMGRPNnM5RTY2Z05UdWVONEMrQ24yTkZMeEhtZVZ0SXpO?=
 =?utf-8?B?bzQxajRFR1h6bDlrZWpjT3lyWTZuNWFvenhnVWdBdDBoY0dEdXlyZ3R5Tnhw?=
 =?utf-8?B?eHNHYmlPSTByOUsrZkw5TFB0eHdscUpwd1JDVGZwbjVKcTdJemhpdVpEcTRt?=
 =?utf-8?B?SkJiMW9rWFVkbE4zN0FRWFRaaE5qZit4OXJCTmREeEo3b3pqTkpPV3NRZkla?=
 =?utf-8?B?T3JNd2dwOEhJZDA2dzl3L29RNk44OE03UDFrOFZaTi80TmgrdTg2c2lKRlB0?=
 =?utf-8?B?eDUxcytZMUZIYVlzbitVSjA5NUhBMEJ5Qnc1OHhFYWJXV1hkQkxZV293UnZ1?=
 =?utf-8?B?aSs2RFlFZnV2M0c2SXBTOHlrVUVGQ295Rjkvb2ZEaWlpb3pNUjRqSmExVUlQ?=
 =?utf-8?B?OHc4YWRXVTF4VityZHNRZ3dUaHQ5UVRRVWtsa2x6VnlhR3I4Zm5UbzdLb21a?=
 =?utf-8?B?MllRVDBqS0ZUb2dScVJuUko4eEswbWM5emJFN0Jra1RMSTZsaURtczA4Zlli?=
 =?utf-8?B?eHdKQlB1aW9lekplYmFETGFSblpTZ2NjdVFUczFRRFJwUWFmSWo0T0FNNGRh?=
 =?utf-8?B?QVI3Rll5eCtpYjR5NDgwblFFR2xWbFU3RG9pVWhlNm84QWZWUXFYQXFLcndp?=
 =?utf-8?B?dVE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 17fb6b2d-a6d0-41b5-f332-08dc6bc93f4c
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 May 2024 23:32:07.5491
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: SjE+aaEXOY3ReJ+SBfHA9Z1++cwqWMQ+lr5oMW16C4222vxq1vyk22/0FgK/AwzKZLq2MHN5fzZsVSBGn7Zeu2TpPq35u+uKbYydYPDDH+M=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB6618
X-OriginatorOrg: intel.com

Hi Babu,

On 3/28/2024 6:06 PM, Babu Moger wrote:
> ABMC (Bandwidth Monitoring Event Configuration) counters can be configured
> by writing to L3_QOS_ABMC_CFG MSR. When ABMC is enabled, the user can
> configure a counter by writing to L3_QOS_ABMC_CFG setting the CfgEn field
> while specifying the Bandwidth Source, Bandwidth Types, and Counter
> Identifier. Add the MSR definition and individual field definitions.
> 
> MSR L3_QOS_ABMC_CFG (C000_03FDh) definitions.
> 
> ==========================================================================
> Bits 	Mnemonic	Description		Access Type   Reset Value
> ==========================================================================
> 63 	CfgEn 		Configuration Enable 	R/W 		0
> 
> 62 	CtrEn 		Counter Enable 		R/W 		0
> 
> 61:53 	– 		Reserved 		MBZ 		0
> 
> 52:48 	CtrID 		Counter Identifier	R/W		0
> 
> 47 	IsCOS		BwSrc field is a COS	R/W		0
> 			(not an RMID)
> 
> 46:44 	–		Reserved		MBZ		0
> 
> 43:32	BwSrc		Bandwidth Source	R/W		0
> 			(RMID or COS)
> 
> 31:0	BwType		Bandwidth types to	R/W		0
> 			track for this counter
> ==========================================================================
> 
> The feature details are documentd in the APM listed below [1].
> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
> Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth
> Monitoring (ABMC).

This changelog is purely a summary of the hardware architecture. I have not come
across a clear explanation on how this architecture is intended to be supported
by resctrl. When would resctrl need/want to set particular fields? What is
the mapping to resctrl?

> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> 
> ---
> v3: No changes.
> v2: No changes.
> ---
>  arch/x86/include/asm/msr-index.h       |  1 +
>  arch/x86/kernel/cpu/resctrl/internal.h | 23 +++++++++++++++++++++++
>  2 files changed, 24 insertions(+)
> 
> diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
> index f16ee50b1a23..ab01abfab089 100644
> --- a/arch/x86/include/asm/msr-index.h
> +++ b/arch/x86/include/asm/msr-index.h
> @@ -1166,6 +1166,7 @@
>  #define MSR_IA32_SMBA_BW_BASE		0xc0000280
>  #define MSR_IA32_EVT_CFG_BASE		0xc0000400
>  #define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
> +#define MSR_IA32_L3_QOS_ABMC_CFG	0xc00003fd
>  
>  /* MSR_IA32_VMX_MISC bits */
>  #define MSR_IA32_VMX_MISC_INTEL_PT                 (1ULL << 14)
> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
> index b559b3a4555e..41b06d46ea74 100644
> --- a/arch/x86/kernel/cpu/resctrl/internal.h
> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
> @@ -106,6 +106,9 @@ cpumask_any_housekeeping(const struct cpumask *mask, int exclude_cpu)
>  #define ASSIGN_TOTAL			BIT(0)
>  #define ASSIGN_LOCAL			BIT(1)
>  
> +/* Maximum assignable counters per resctrl group */
> +#define MAX_ASSIGN_CNTRS		2
> +
>  struct rdt_fs_context {
>  	struct kernfs_fs_context	kfc;
>  	bool				enable_cdpl2;
> @@ -210,6 +213,7 @@ enum rdtgrp_mode {
>   * @crdtgrp_list:		child rdtgroup node list
>   * @rmid:			rmid for this rdtgroup
>   * @mon_state:			Assignment state of the group
> + * @abmc_ctr_id:		ABMC counterids assigned to this group
>   */
>  struct mongroup {
>  	struct kernfs_node	*mon_data_kn;
> @@ -217,6 +221,7 @@ struct mongroup {
>  	struct list_head	crdtgrp_list;
>  	u32			rmid;
>  	u32			mon_state;
> +	u32			abmc_ctr_id[MAX_ASSIGN_CNTRS];
>  };
>  
>  /**
> @@ -566,6 +571,24 @@ union cpuid_0x10_x_edx {
>  	unsigned int full;
>  };
>  
> +/*
> + * L3_QOS_ABMC_CFG MSR details. ABMC counters can be configured
> + * by writing to L3_QOS_ABMC_CFG.

There are many fields in this structure ... how is resctrl expected
to set these fields in order to configure a counter? Please expand the
comments.

> + */
> +union l3_qos_abmc_cfg {
> +	struct {
> +		unsigned long  bw_type	:32,
> +			       bw_src	:12,
> +			       rsvrd1	: 3,

Considering how "reserved" is spelled it is
unexpected to see "rsvrd"


> +			       is_cos	: 1,
> +			       ctr_id	: 5,
> +			       rsvrd	: 9,
> +			       ctr_en	: 1,
> +			       cfg_en	: 1;
> +	} split;
> +	unsigned long full;
> +};
> +
>  void rdt_last_cmd_clear(void);
>  void rdt_last_cmd_puts(const char *s);
>  __printf(1, 2)


Reinette