Received-SPF: pass (google.com: domain of linux-kernel+bounces-214191-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <31497c62-fe32-43a6-ba8d-361c4c89422b@intel.com>
Date: Thu, 13 Jun 2024 18:44:54 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v4 13/19] x86/resctrl: Add data structures for ABMC
 assignment
To: Babu Moger <babu.moger@amd.com>, <corbet@lwn.net>, <fenghua.yu@intel.com>,
	<tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>,
	<dave.hansen@linux.intel.com>
CC: <x86@kernel.org>, <hpa@zytor.com>, <paulmck@kernel.org>,
	<rdunlap@infradead.org>, <tj@kernel.org>, <peterz@infradead.org>,
	<yanjiewtw@gmail.com>, <kim.phillips@amd.com>, <lukas.bulwahn@gmail.com>,
	<seanjc@google.com>, <jmattson@google.com>, <leitao@debian.org>,
	<jpoimboe@kernel.org>, <rick.p.edgecombe@intel.com>,
	<kirill.shutemov@linux.intel.com>, <jithu.joseph@intel.com>,
	<kai.huang@intel.com>, <kan.liang@linux.intel.com>,
	<daniel.sneddon@linux.intel.com>, <pbonzini@redhat.com>,
	<sandipan.das@amd.com>, <ilpo.jarvinen@linux.intel.com>,
	<peternewman@google.com>, <maciej.wieczor-retman@intel.com>,
	<linux-doc@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
	<eranian@google.com>, <james.morse@arm.com>
References: <cover.1716552602.git.babu.moger@amd.com>
 <a68ff64d52eff2de91a3c6030c2650396ea7fb8c.1716552602.git.babu.moger@amd.com>
From: Reinette Chatre <reinette.chatre@intel.com>
Content-Language: en-US
In-Reply-To: <a68ff64d52eff2de91a3c6030c2650396ea7fb8c.1716552602.git.babu.moger@amd.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?M3hiUUVkR3V6T0FWbXhKRDNTclN4OGlJRUl6MlpPcEJCbEd3LzBUWldKL29h?=
 =?utf-8?B?YmRHcnRBYUNCNGFRTEw1YkQ2UWxBeWZDVmoySGFnOVQ0RGRWamh6REZBZWZZ?=
 =?utf-8?B?cGU3Uk1vb2xyWEp4T0Rja3k5VGtRSDhRT3N1YmZLTTZiUkFlalBvMmttNVcx?=
 =?utf-8?B?Ry84c3dFeEczbm01eXpuZTl3aUk3amo2NTBBb1dpbU12MXM4bjN1N2p4eVJ2?=
 =?utf-8?B?THFRZHc4QXcwYTNrSGxQZGppZ1pXZTc1TnhRV2E3SmxuUFFrMWd1YjFHc0Ns?=
 =?utf-8?B?WTdtY2dlOFFncjNpM0E2WVZOOEZJcjlqenNkZmM3d0pGWk9WWHh4cU1ET0ZF?=
 =?utf-8?B?ZnFXUEtMdDhuL2pTempYdmlhbXJDN1c2N2hHYkZSSEdJTEdhdldWUXZ1eDZP?=
 =?utf-8?B?ZEpwdTN1MnFzclh4V1RNNVkrQmVqdkRCbVkzS2gwbTF3SlVBSldYRGhVTXpJ?=
 =?utf-8?B?ckVYby9GR3hBZjJ2UENrR3AxVTFOZ3RHTjlDN3dMMzhXZXVxUjR5OTM2QTZJ?=
 =?utf-8?B?RmMxWjU3bWtpS2s1Z0pHcm9rdEh0c25WTVA0b25LZklWbFd6SEM2K25POFRZ?=
 =?utf-8?B?TklLMFBMTngvU0JFVzFXekZXQnBOMDQ5YUlMMmlYNklZL1BHS01pdmRvTUs0?=
 =?utf-8?B?OWlkR29OdmYrM3Z2MEpEbWswc2YwNzR0VndDL3p2c2VyZWFUTUEvRlpDeHVC?=
 =?utf-8?B?dDI2MWhPZ0Zza01IYlZCU0VIV3VwZTJqUjhFRnMrTHErdjhScWxtMkVSQTdl?=
 =?utf-8?B?OTZVc3pIb05UV3FBdTlPVEo3VnZZRDR3SGhTUWpFNjZlSzBpbjVnQm1HNVVC?=
 =?utf-8?B?bG9DZ3VVRlpUZE5Ia2J2UnpyY01JeURmTmxrOS9PZGZMZW1MdVhZNXQyb1VG?=
 =?utf-8?B?MVlYTkhZVVZqbWpLL0MvdU9KajhNejI1NHc2Vy80NGpDcXFESDhBb3VXS0Zi?=
 =?utf-8?B?MHpvT081TWVqZ0JlV2RpZXVISzZ5RTQwV2ZKZWtlRWdSTzZveUcyNjZaUjRt?=
 =?utf-8?B?OGh1VVNpVDZJR0hJeFZtUi9DUWtnNzg5ZVVaWXg5TU41L2d1ZDRYS0VTRWpN?=
 =?utf-8?B?Q2VwSGJQNFNET1A0TnNkQW84RGZqV0t3aW0vRTQxdUFKamZ4dGhlWkdFTytl?=
 =?utf-8?B?L2w2Q2IxYlVkbThlNjJyWk5wTER1cHlFS1J5NnpoZXVPRzBXMEI3QlpRWVFN?=
 =?utf-8?B?dncwd2JrRC9ESHFpUnhlUWdrb0pPNTJkYTdnVWVVSWRFRStJb1ZRbmZSMzZh?=
 =?utf-8?B?TWFiaU9OYkhUTEZ4OUxIWUswVi81Qk9BUzRGTURscllZdFNCQVcxNkZNU1ky?=
 =?utf-8?B?VnFTTHBaeWdPYjd0by9uYWs0dzNFZ1ZqTGhXcXcxNWpYdkFUYUx0QXhZN0RB?=
 =?utf-8?B?aVVMWGdqZEcvMk53SGg4K1EyM1JOeXpQRXMyeUZ4MUdtb3FEMmQ4U0NGRGVn?=
 =?utf-8?B?amYwTWxIQTBtQjlRU0g3TTZ4VnNMMmpyNjN0S2FQRGxQQ2JnUElDaUtaZFd4?=
 =?utf-8?B?Y0R0Znk1YU9QbWYyUFhvQ1BZeklvNEF1bmwxR3pqb09nR09IZytyMzRMVzNR?=
 =?utf-8?B?a0Z0aFFlRlRUYlNFeEx6d0ZuTnFndDRxeTVybW9mOFhzUjROYjdpMFRMWGhT?=
 =?utf-8?B?UDhvVTRDeUUwTm1pMUFPTzVyeUpOTmhDUnR2WFZUblRvUlU4Mm5obmY3RmIy?=
 =?utf-8?B?YlZscDg0RDhiOVBLbDFMOFh5a1FDdi9XVkZxVjJEQkdlK3J0M0ExSjQ0b3dS?=
 =?utf-8?B?akRmc2J2ZzN0V09hQ3h2VGdXOHpzbTdUazRTVCtONklER2xhRHdRZ0t6Zk9m?=
 =?utf-8?B?dDBHaGhUeDliOEpQbTBnUzkyMGRVN0M1cG8vWnUzZjh6eFcvZlBDbjNrSDdl?=
 =?utf-8?B?WE1lT2xhVHIxYmdmMVh6Zm9MRkxtSGxkNllCUDZrUHdDQ1E4WmlsRlhEMTh1?=
 =?utf-8?B?R3psSEk4aUg1U2Y5WHR2TFQ5Z1cybnBwVStrNU9FcW53UTA2QTdoMkMvdmV6?=
 =?utf-8?B?emYwUk1UWVdCQW4wL1A3OTZBZVNmYjN5ZVA3Mm9SQndzUFFRMmRuQWhUSUFD?=
 =?utf-8?B?T0hlaHE4bEk4TGhYc0dnT3ZMSGNPaFBFN1VzODYzOW1JajQzaCtXNG9yZ0o0?=
 =?utf-8?B?TUdoWXMyUU1ZRld2OGZRM2JWM3REVXFRenBLSllHMFo1MXYvMW5LaGgraml3?=
 =?utf-8?B?M3c9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 7dc3003a-b6bd-4930-d970-08dc8c139c40
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jun 2024 01:45:03.4866
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: zHjis9ZIFBi5jhPCdR8uY8gFGxT/NV6bjZ+G6itVGYXAbRQEPzT9TI3VhMh2ndqLIikaEwlR2t6V8jdTMs8NAYQUFoCRljCltQbUoi/8DfE=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR11MB6203
X-OriginatorOrg: intel.com

Hi Babu,

On 5/24/24 5:23 AM, Babu Moger wrote:
> The ABMC feature provides an option to the user to assign a hardware
> counter to an RMID and monitor the bandwidth as long as the counter
> is assigned. The bandwidth events will be tracked by the hardware until
> the user changes the configuration. Each resctrl group can configure
> maximum two counters, one for total event and one for local event.
> 
> The counters are configured by writing to MSR L3_QOS_ABMC_CFG.
> Configuration is done by setting the counter id, bandwidth source (RMID),
> bandwidth types. Reading L3_QOS_ABMC_DSC returns the configuration of

What is a "bandwidth type" here?

> the counter id specified in L3_QOS_ABMC_CFG.
> 
> Attempts to read or write these MSRs when ABMC is not enabled will result
> in a #GP(0) exception.
> 
> MSR L3_QOS_ABMC_CFG (0xC000_03FDh) and L3_QOS_ABMC_DSC (0xC000_03FEh)
> details.
> ==========================================================================
> Bits 	Mnemonic	Description		Access Type   Reset Value
> ==========================================================================
> 63 	CfgEn 		Configuration Enable 	R/W 		0
> 
> 62 	CtrEn 		Counter Enable 		R/W 		0
> 
> 61:53 	– 		Reserved 		MBZ 		0
> 
> 52:48 	CtrID 		Counter Identifier	R/W		0
> 
> 47 	IsCOS		BwSrc field is a COS	R/W		0
> 			(not an RMID)
> 
> 46:44 	–		Reserved		MBZ		0
> 
> 43:32	BwSrc		Bandwidth Source	R/W		0
> 			(RMID or COS)
> 
> 31:0	BwType		Bandwidth types to	R/W		0
> 			track for this counter
> ==========================================================================
> 

This is copy&paste from AMD spec but needs some translation to map it to
resctrl.

> The feature details are documented in the APM listed below [1].
> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
> Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth
> Monitoring (ABMC).
> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> ---
> v4: Added more descriptions.
>      Changed the name abmc_ctr_id to ctr_id.
>      Added L3_QOS_ABMC_DSC. Used for reading the configuration.
> 
> v3: No changes.
> 
> v2: No changes.
> ---
>   arch/x86/include/asm/msr-index.h       |  2 ++
>   arch/x86/kernel/cpu/resctrl/internal.h | 30 ++++++++++++++++++++++++++
>   2 files changed, 32 insertions(+)
> 
> diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
> index 5f9a0139e98c..6d2fe39ac68f 100644
> --- a/arch/x86/include/asm/msr-index.h
> +++ b/arch/x86/include/asm/msr-index.h
> @@ -1172,6 +1172,8 @@
>   #define MSR_IA32_SMBA_BW_BASE		0xc0000280
>   #define MSR_IA32_EVT_CFG_BASE		0xc0000400
>   #define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
> +#define MSR_IA32_L3_QOS_ABMC_CFG	0xc00003fd
> +#define MSR_IA32_L3_QOS_ABMC_DSC	0xc00003fe
>   
>   /* MSR_IA32_VMX_MISC bits */
>   #define MSR_IA32_VMX_MISC_INTEL_PT                 (1ULL << 14)
> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
> index 60a1ca0a11a7..45ed33f4f0ff 100644
> --- a/arch/x86/kernel/cpu/resctrl/internal.h
> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
> @@ -107,6 +107,9 @@ cpumask_any_housekeeping(const struct cpumask *mask, int exclude_cpu)
>   #define ASSIGN_TOTAL			BIT(QOS_L3_MBM_TOTAL_EVENT_ID)
>   #define ASSIGN_LOCAL			BIT(QOS_L3_MBM_LOCAL_EVENT_ID)
>   
> +/* Maximum assignable counters per resctrl group */
> +#define MAX_CNTRS		2
> +
>   struct rdt_fs_context {
>   	struct kernfs_fs_context	kfc;
>   	bool				enable_cdpl2;
> @@ -211,6 +214,7 @@ enum rdtgrp_mode {
>    * @crdtgrp_list:		child rdtgroup node list
>    * @rmid:			rmid for this rdtgroup
>    * @mon_state:			Assignment state of the group
> + * @ctr_id:			ABMC counter ids assigned to this group
>    */
>   struct mongroup {
>   	struct kernfs_node	*mon_data_kn;
> @@ -218,6 +222,7 @@ struct mongroup {
>   	struct list_head	crdtgrp_list;
>   	u32			rmid;
>   	u32			mon_state;
> +	u32			ctr_id[MAX_CNTRS];

To match "num_cntrs" it seems "cntr_id" may be more appropriate?

>   };
>   
>   /**
> @@ -568,6 +573,31 @@ union cpuid_0x10_x_edx {
>   	unsigned int full;
>   };
>   
> +/*
> + * ABMC counters can be configured  by writing to L3_QOS_ABMC_CFG.

extra space

> + * @bw_type		: Bandwidth types to track for this counter

What is a "Bandwidth type"? What are possible values?

> + * @bw_src		: Bandwidth Source (RMID or CLOSID)
> + * @reserved1		: Reserved
> + * @is_cos		: BwSrc field is a COS (not an RMID)

Please be consistent wrt "CLOS"/"CLOSID" in comment and member name

> + * @ctr_id		: Counter Identifier

Please be consistent with "cntr" vs "ctr". Earlier the series
introduces "num_cntrs" so cntr seems appropriate?

> + * @reserved		: Reserved
> + * @ctr_en		: Counter Enable bit
> + * @cfg_en		: Configuration Enable bit

Why are there two enable bits? How are they used?

(please match other kernel doc wrt : usage)

> + */
> +union l3_qos_abmc_cfg {
> +	struct {
> +		unsigned long  bw_type	:32,
> +			       bw_src	:12,
> +			       reserved1: 3,
> +			       is_cos	: 1,
> +			       ctr_id	: 5,
> +			       reserved	: 9,
> +			       ctr_en	: 1,
> +			       cfg_en	: 1;
> +	} split;
> +	unsigned long full;
> +};
> +
>   void rdt_last_cmd_clear(void);
>   void rdt_last_cmd_puts(const char *s);
>   __printf(1, 2)

Reinette