Received-SPF: pass (google.com: domain of linux-kernel+bounces-172156-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <b367a6ba-3ff1-4173-bf71-cd7a04bf9882@amd.com>
Date: Tue, 7 May 2024 15:40:52 -0500
User-Agent: Mozilla Thunderbird
Reply-To: babu.moger@amd.com
Subject: Re: [RFC PATCH v3 10/17] x86/resctrl: Add data structures for ABMC
 assignment
To: Reinette Chatre <reinette.chatre@intel.com>, corbet@lwn.net,
 fenghua.yu@intel.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de,
 dave.hansen@linux.intel.com
Cc: x86@kernel.org, hpa@zytor.com, paulmck@kernel.org, rdunlap@infradead.org,
 tj@kernel.org, peterz@infradead.org, yanjiewtw@gmail.com,
 kim.phillips@amd.com, lukas.bulwahn@gmail.com, seanjc@google.com,
 jmattson@google.com, leitao@debian.org, jpoimboe@kernel.org,
 rick.p.edgecombe@intel.com, kirill.shutemov@linux.intel.com,
 jithu.joseph@intel.com, kai.huang@intel.com, kan.liang@linux.intel.com,
 daniel.sneddon@linux.intel.com, pbonzini@redhat.com, sandipan.das@amd.com,
 ilpo.jarvinen@linux.intel.com, peternewman@google.com,
 maciej.wieczor-retman@intel.com, linux-doc@vger.kernel.org,
 linux-kernel@vger.kernel.org, eranian@google.com, james.morse@arm.com
References: <cover.1711674410.git.babu.moger@amd.com>
 <c3103dd30bc07070380a01922f66208759afe837.1711674410.git.babu.moger@amd.com>
 <3149b62a-b9a4-49c6-a757-5a0e36df62df@intel.com>
Content-Language: en-US
From: "Moger, Babu" <babu.moger@amd.com>
In-Reply-To: <3149b62a-b9a4-49c6-a757-5a0e36df62df@intel.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?NCtTbmd0TzEvbVNtTm92MUpsL3FDT2lBdFIzTUt1UGVUMlJsRDMxTEhFcmgw?=
 =?utf-8?B?QU1hejZlRys4TG9vcy9FV2tLenhLMk56NWEvcWU5eVpZZnZvT2hRZ2F0bDRx?=
 =?utf-8?B?SUNzUFJPa3pZQXNpSXFsOHlpRG8zaklBWFo3TmpML1dNSFFiVEkvNU84NGZv?=
 =?utf-8?B?dUNqWWY0TGdVbG9CcTF4S3ZzZm1oS2lIZWlWQlBLbWFDK2hFUEh6cEY4emlM?=
 =?utf-8?B?eWQ1V1FkNnRoRUtiOE5aOFhoMXNTVmVCL1FLTnU1VnE3d3RPaXhOd2FMT3I2?=
 =?utf-8?B?YWdYWGQwQ2IyWTdlUDcxTDFCTld2ZHlTbnVMUFBKemQ0dGRXK0NMRGpSbWNn?=
 =?utf-8?B?MnZPaVlIMTVZbWR4azBrUjYyMVpLNWNBcCtoV0I4ZmRQeWFTd0l1T252SUZ5?=
 =?utf-8?B?VGx3V2VVSkQwTzdMR0JWSWJYNkZIMFk0S3NMMXhHVGQ0WHZISklKUkduaHp3?=
 =?utf-8?B?QkNmRUdMU05tRDFnWUhtM1Z2aUFvWmM1N083MUlLNWIxQnVZb0RzdFFGbWZx?=
 =?utf-8?B?YXZlUCs5R0N2cHVJdjd0NjREVndFUktta0Era0lVRWxEdXcwRDFWOFYvbG5X?=
 =?utf-8?B?RXpsblFhNXZ0amFOUUFpNk1FeHV0NU9FWHV4ODJ0Ni9wcWpGNnNESG9xK0JQ?=
 =?utf-8?B?YzdpdzVyaDltU3lzYW9MU2JIaW96RjE4dnZwS0NFSWZwemtPWHdqQlF1Wm5H?=
 =?utf-8?B?Mng5aVRtYmpGVTVNM2FMcnBxb04yWnROR2J3QVMrck9WRmdzalBsTFhEeGp1?=
 =?utf-8?B?S3hCMjIxYkZWSFBOek1Mclc4MWpaNVdqYzRvSTh2bXdSZ0FVQjdVamk2S2hl?=
 =?utf-8?B?TlJlYWRTS24rcFdyejBaTk9ta2tndG1RUUw5ZCsxa0FINjlVSzYzRFBBUHRI?=
 =?utf-8?B?OU9IZG9selMzRlZNcmJ1UE5Za2RjWUp4Z2lXZXhqVHJWUTlDYmczeWxtYmpQ?=
 =?utf-8?B?OGdwaTUyK0pxaHhjUkFZYnNyM0Q2MkhHMDRaSXRQOGJCaGZSVXl5N25EM2NY?=
 =?utf-8?B?MVpGZ2RQaTlSODI1Tlg4YWdYZVFkTTBmbXQzQUhlcHl4NzYyNW9vanNCZGJ1?=
 =?utf-8?B?TkxZOEhzYmlUM2xjQ2VCK29aSDJQQkFIUVpSZ1MzSll0Ujd4cmxoN0dJQjli?=
 =?utf-8?B?c2x4WGRQTDkwYllEZzZFVFRvZStTRzJKRjFTVEE2N2RpamJKdkN1ZVJGWDB3?=
 =?utf-8?B?MjdXbkRVcCsxcWdTN1RnQWFKdkJUZmJZRFNzbzh5ZmJWeXA2UzRPQmxINC9t?=
 =?utf-8?B?Z2pRRmNwMzh0UkFqaGRocWhEaEluUGZ6akJFM1IzY2oxejRwWmwrWDZxWTVI?=
 =?utf-8?B?Y2RpbkR1N20rQ1ZGQm9NSlR4c3BLVlR3REJnTlMyNE8xVkR1Y1ZLaXAyeW1G?=
 =?utf-8?B?THJqbXFkRTVUV3NQUGQ5U1hsVGFydXV0aHB3TjV4Ym9RK1pBNmNobUFJNXh2?=
 =?utf-8?B?NXEzZTlzbnFYWFFIYXA2QWJ0WGRaSnpCVzBJS2c1bFF0SEN3QWVKUjMrNStK?=
 =?utf-8?B?bGxEamNxUVQvcmxnSmprSzMzUFBpREw0aTZ3Z3hYZ0txU1N0dnpHWS9BN3dO?=
 =?utf-8?B?cHhuNi9TaEIrZFdCME1wZWlNdTlMWGNBbzAxMWNNVG82TitSZHRzaFpDT2Ns?=
 =?utf-8?B?OHltRVVHb2NNdG82KzhzMzBNbzVOTTloR0ZqTzFERlBoeUhrVHgvejRVOXJY?=
 =?utf-8?B?UDNTVkgwdXVoNkx4eEtXSHJzUjhZVkRBWkErZEN5ZS9JSWtRY1BDcWFhdmdK?=
 =?utf-8?B?b2JjOEdYekFLNUZQVkd6YUQ5K3YwVnRYZWM1Q1BGcXEyeUFWMFkzYklDVFhC?=
 =?utf-8?B?OUJ2QkZOT1JML0N0VjFRT2VKNGpOMHlYeXQ4OHlRdFNPT09MSFBOYWlOUmt2?=
 =?utf-8?B?Qjd0MXd3ZXpGM3RnNmxJOHZCYXVLekJmcEhUTEFSclZ1QkFuSUlUa2JpZnlx?=
 =?utf-8?B?Q2JHTFIybnVDRE9Rb0p6L3d5ai9yVUsxc3duWUpxNWVSNmF5eVNjOHpEaE81?=
 =?utf-8?B?N2JCOEYzMmNUcXQ1bUpVTm83R3NCYnJLRE9iOXFXeUU3VXJmVSt2KzVyMHlC?=
 =?utf-8?B?TTIyVEQ2OUc0MWFrbjV6L1dvM2kxTTd4MnloaDJRWlRTSVBzL0ZVblZwa3or?=
 =?utf-8?Q?JTdY=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9d343bce-1725-4662-78ab-08dc6ed5fefa
X-MS-Exchange-CrossTenant-AuthSource: MW3PR12MB4553.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 May 2024 20:40:56.5919
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ybaZsoooxyAtIT7XfN169MLOR5nbov4rhba1COQYxyER/4O8D/Zjy+0DaSUIoZzT
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9401

Hi Reinette,


On 5/3/24 18:32, Reinette Chatre wrote:
> Hi Babu,
> 
> On 3/28/2024 6:06 PM, Babu Moger wrote:
>> ABMC (Bandwidth Monitoring Event Configuration) counters can be configured
>> by writing to L3_QOS_ABMC_CFG MSR. When ABMC is enabled, the user can
>> configure a counter by writing to L3_QOS_ABMC_CFG setting the CfgEn field
>> while specifying the Bandwidth Source, Bandwidth Types, and Counter
>> Identifier. Add the MSR definition and individual field definitions.
>>
>> MSR L3_QOS_ABMC_CFG (C000_03FDh) definitions.
>>
>> ==========================================================================
>> Bits 	Mnemonic	Description		Access Type   Reset Value
>> ==========================================================================
>> 63 	CfgEn 		Configuration Enable 	R/W 		0
>>
>> 62 	CtrEn 		Counter Enable 		R/W 		0
>>
>> 61:53 	– 		Reserved 		MBZ 		0
>>
>> 52:48 	CtrID 		Counter Identifier	R/W		0
>>
>> 47 	IsCOS		BwSrc field is a COS	R/W		0
>> 			(not an RMID)
>>
>> 46:44 	–		Reserved		MBZ		0
>>
>> 43:32	BwSrc		Bandwidth Source	R/W		0
>> 			(RMID or COS)
>>
>> 31:0	BwType		Bandwidth types to	R/W		0
>> 			track for this counter
>> ==========================================================================
>>
>> The feature details are documentd in the APM listed below [1].
>> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
>> Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth
>> Monitoring (ABMC).
> 
> This changelog is purely a summary of the hardware architecture. I have not come
> across a clear explanation on how this architecture is intended to be supported
> by resctrl. When would resctrl need/want to set particular fields? What is
> the mapping to resctrl?

Something like this in the changelog?

ABMC feature provides an option to assign(or pin) an RMID to the
hardware counter and monitor the bandwidth for a longer duration.

Hardware counters can be configured by writing to L3_QOS_ABMC_CFG MSR.
Configuration is done by setting the CfgEn field while specifying the
Bandwidth Source(RMID or CLOS), Bandwidth Types, and Counter Identifier.

Add the configuration register definition.

> 
>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
>>
>> ---
>> v3: No changes.
>> v2: No changes.
>> ---
>>  arch/x86/include/asm/msr-index.h       |  1 +
>>  arch/x86/kernel/cpu/resctrl/internal.h | 23 +++++++++++++++++++++++
>>  2 files changed, 24 insertions(+)
>>
>> diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
>> index f16ee50b1a23..ab01abfab089 100644
>> --- a/arch/x86/include/asm/msr-index.h
>> +++ b/arch/x86/include/asm/msr-index.h
>> @@ -1166,6 +1166,7 @@
>>  #define MSR_IA32_SMBA_BW_BASE		0xc0000280
>>  #define MSR_IA32_EVT_CFG_BASE		0xc0000400
>>  #define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
>> +#define MSR_IA32_L3_QOS_ABMC_CFG	0xc00003fd
>>  
>>  /* MSR_IA32_VMX_MISC bits */
>>  #define MSR_IA32_VMX_MISC_INTEL_PT                 (1ULL << 14)
>> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
>> index b559b3a4555e..41b06d46ea74 100644
>> --- a/arch/x86/kernel/cpu/resctrl/internal.h
>> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
>> @@ -106,6 +106,9 @@ cpumask_any_housekeeping(const struct cpumask *mask, int exclude_cpu)
>>  #define ASSIGN_TOTAL			BIT(0)
>>  #define ASSIGN_LOCAL			BIT(1)
>>  
>> +/* Maximum assignable counters per resctrl group */
>> +#define MAX_ASSIGN_CNTRS		2
>> +
>>  struct rdt_fs_context {
>>  	struct kernfs_fs_context	kfc;
>>  	bool				enable_cdpl2;
>> @@ -210,6 +213,7 @@ enum rdtgrp_mode {
>>   * @crdtgrp_list:		child rdtgroup node list
>>   * @rmid:			rmid for this rdtgroup
>>   * @mon_state:			Assignment state of the group
>> + * @abmc_ctr_id:		ABMC counterids assigned to this group
>>   */
>>  struct mongroup {
>>  	struct kernfs_node	*mon_data_kn;
>> @@ -217,6 +221,7 @@ struct mongroup {
>>  	struct list_head	crdtgrp_list;
>>  	u32			rmid;
>>  	u32			mon_state;
>> +	u32			abmc_ctr_id[MAX_ASSIGN_CNTRS];
>>  };
>>  
>>  /**
>> @@ -566,6 +571,24 @@ union cpuid_0x10_x_edx {
>>  	unsigned int full;
>>  };
>>  
>> +/*
>> + * L3_QOS_ABMC_CFG MSR details. ABMC counters can be configured
>> + * by writing to L3_QOS_ABMC_CFG.
> 
> There are many fields in this structure ... how is resctrl expected
> to set these fields in order to configure a counter? Please expand the
> comments.

Sure.

> 
>> + */
>> +union l3_qos_abmc_cfg {
>> +	struct {
>> +		unsigned long  bw_type	:32,
>> +			       bw_src	:12,
>> +			       rsvrd1	: 3,
> 
> Considering how "reserved" is spelled it is
> unexpected to see "rsvrd"

Will change it to "reserved1" and "reserved" (below).

> 
> 
>> +			       is_cos	: 1,
>> +			       ctr_id	: 5,
>> +			       rsvrd	: 9,
>> +			       ctr_en	: 1,
>> +			       cfg_en	: 1;
>> +	} split;
>> +	unsigned long full;
>> +};
>> +
>>  void rdt_last_cmd_clear(void);
>>  void rdt_last_cmd_puts(const char *s);
>>  __printf(1, 2)
> 
> 
> Reinette
> 

-- 
Thanks
Babu Moger