Received-SPF: pass (google.com: domain of linux-kernel+bounces-172044-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Message-ID: <54ec73f9-6c9e-44f4-8ee9-a683bfcee607@amd.com>
Date: Tue, 7 May 2024 14:12:31 -0500
User-Agent: Mozilla Thunderbird
Reply-To: babu.moger@amd.com
Subject: Re: [RFC PATCH v3 07/17] x86/resctrl: Add support to enable/disable
 ABMC feature
To: Reinette Chatre <reinette.chatre@intel.com>, corbet@lwn.net,
 fenghua.yu@intel.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de,
 dave.hansen@linux.intel.com
Cc: x86@kernel.org, hpa@zytor.com, paulmck@kernel.org, rdunlap@infradead.org,
 tj@kernel.org, peterz@infradead.org, yanjiewtw@gmail.com,
 kim.phillips@amd.com, lukas.bulwahn@gmail.com, seanjc@google.com,
 jmattson@google.com, leitao@debian.org, jpoimboe@kernel.org,
 rick.p.edgecombe@intel.com, kirill.shutemov@linux.intel.com,
 jithu.joseph@intel.com, kai.huang@intel.com, kan.liang@linux.intel.com,
 daniel.sneddon@linux.intel.com, pbonzini@redhat.com, sandipan.das@amd.com,
 ilpo.jarvinen@linux.intel.com, peternewman@google.com,
 maciej.wieczor-retman@intel.com, linux-doc@vger.kernel.org,
 linux-kernel@vger.kernel.org, eranian@google.com, james.morse@arm.com
References: <cover.1711674410.git.babu.moger@amd.com>
 <0db75c94886da62b8da498ef159d8fe27b0b3811.1711674410.git.babu.moger@amd.com>
 <54b1fe8f-13e5-440b-bb36-4100c1d283d0@intel.com>
Content-Language: en-US
From: "Moger, Babu" <babu.moger@amd.com>
In-Reply-To: <54b1fe8f-13e5-440b-bb36-4100c1d283d0@intel.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?Ykk1cS9nV0FOSG40Q2FSVkJ2Z0MwUlZ5RFhReDlWOStyeWRscFhYWU10N3Y3?=
 =?utf-8?B?YUdNNXRNSC9td0Y1WU8yZ3FwTkdFNHpRZ2JWYkIrQ2F6VEtNWmpvMUJUdGFj?=
 =?utf-8?B?cTFBQXVJTU53TllrVElqVTU5d1NDM0RYa0tYQ3UrQkRUblNnSGppMTlRbkIw?=
 =?utf-8?B?b2dsQkZ0cFYvVzhKeitZN0NaK1g0SnlxUXV2ZXBQdEc3QmNTSDNBWlRsSnM3?=
 =?utf-8?B?VTlRNDVYYTRJNzZzNnZJdVFZMzAyRFFRUVcwSWNTWVVIdDc1clFOVXRSemV4?=
 =?utf-8?B?RWplZHJtR3U4Z3pzaTB6WVZTaWJSd2NmNVRjNno3bitiUFd5WUF2OE1ndmRh?=
 =?utf-8?B?eVcvOXhxTXVoUHo1TGdxeUJlUmdsNFo5RithU0hIV0lUbGhTOHBoVGtzM1F6?=
 =?utf-8?B?RFluVTNaVzVsQ1QwVy9nM1p2ckdUWkVKVm5zUzNIMXVNTU5pSEc3ZHRvQjJn?=
 =?utf-8?B?eFBGMzdUY21qZ2sxSG1hR1BNUWYzbXNMT285cGJ3US9xVkdVcEFMMmpWMVAr?=
 =?utf-8?B?SDZPNWJ2cjdZTmJGd21zd0pQSUJKOHNrSzlIbDVTMlk1Z0xDOW5LS09Sek91?=
 =?utf-8?B?TWExUWc0dm9UZzRKZUhSOVUzdUlrdFBGOEVncUM4aTY0U01Yd3lsVjJKa21X?=
 =?utf-8?B?Y0JUd0xHZ3NFZ09GeCtVYU4yKzA1L21MSkdFMjE2RStJWmc2TjFwZkM5YkJC?=
 =?utf-8?B?UTl5NWl4eFlBcUc0T1MvWUZObFdndUd0OUhkN1h0WEliUzJCMXVZQzBZRWkz?=
 =?utf-8?B?R0h6Ync2dmJWZjdMZmswN3RBZHhnemZkN2xhSE1PU0FtbnpWV3dWWGJKVHJ0?=
 =?utf-8?B?V2wxZDl0WFNKVFJRQStsRTgwUFZyY3ErVnhqb1dJQXN5LzlPMzZrdVo1K0JZ?=
 =?utf-8?B?V0xLTnYxSnpSaVcyRmRyUWZ5UEpWeXNSbW96Q2t5L2JGVGdMajc5MGY2Q0dt?=
 =?utf-8?B?cWZhejM4V0svV3pzOVNTajVrL2lzNUV1Z0NsN213TlJSMmFQZWJZR05qQWQ0?=
 =?utf-8?B?SjhjVkRYc3BkTVYrdzN0N1pwQ1pPUDQyYnN2QVd1MFV1Y2NtcmowMWxDOCtn?=
 =?utf-8?B?eWxxOGhlN1Q3TlR0ZUZZZkszT0t6MTdyRDVNL1NQaUdVdWxST0lPcTRiTFNU?=
 =?utf-8?B?aysvL2Z5Skd3Y1hEOXVBTURoWkUrZFFTbFRrS0IvWWsrVWNTNHQrSjN0L2Na?=
 =?utf-8?B?RlVTMXlaWTR0MDN2emdIazNZTDBWRUl0bVdnWnIraVEzOXZQa24zUTdhSUtU?=
 =?utf-8?B?NFBwamJFWmsrNUxONDdxQm9qcFdsdTJEU1NkdmhISUdpRmk3WG9HNmhOVEpu?=
 =?utf-8?B?OEh3b0dmdFZlaWgzQkhTa2Z4a1pCQ3hkZFExeTN4cjkzU1EraFdQWGFEbyt0?=
 =?utf-8?B?YmR2OVloQmNKUjR3dWhnL0RWKzgzdjgzTGxhTHNaWm1OVlJLSFFzVVRsOU9S?=
 =?utf-8?B?MFRmNGFDRnVtdUd2MG1ac01Tbnh0OTI5NjlFazJBMTQ0UVdGV0NuZGUrRHFN?=
 =?utf-8?B?VHl2Y2Q0dng4dzJnbURMM0w0cTdUMmJMVmpGYzZvT2xlV2VUVmYzY0kyOVJn?=
 =?utf-8?B?THVUTlU5YjNJSUk2bk9JYWNjbUVhc0RVNlpFUGRiSFlVZHdvbUcxeTVLVEJS?=
 =?utf-8?B?WHZqWGh4K2Z3ZjM2V1hnaWdkb0ZYSFFRRjRLUU5EZGs1Qi9ybklhWU5rUFZm?=
 =?utf-8?B?alhaa2JaUi8yMG5UalB3Mit2S3ZlTzRTTFVWcHRpUFJKRzRDbVhDK0tsYVhZ?=
 =?utf-8?B?TG8xbHlMa2E0M1lYQnFtWVgxcUNneTgzM3NhUlppQVNOWEZJN0lZc1QxWk9R?=
 =?utf-8?B?bVoxc09ub0J4cXBTVkFvYk5JVkxZZEFsVGNnd3p1Mm9vVU83SHM0bFJsZ0lZ?=
 =?utf-8?B?M2VPNm1nbExaV0pCbVprQzNDVjZENGdUai9WNDdUSGNZNUxFbGVmY2wwd0g2?=
 =?utf-8?B?U1d0aDh0WUg3cmJrMmVRZVR6WkhoVW5HeFN2N1FRUDN4OTRkUU1nN3B2bU9m?=
 =?utf-8?B?SEJIazRMK21wQ0dMWlBxSXFqSkMrQjkvU2xrUW5jZVZ5S0JHa2tpVUE3bWty?=
 =?utf-8?B?YmpZM2ZEVUU4WVlOQkI5WFVYOThhRks1N3RDUm05aEJRKzlQWlh5Z04xeGZS?=
 =?utf-8?Q?0Vjs=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: cbf95d6c-2f81-44bc-dca5-08dc6ec9a7ee
X-MS-Exchange-CrossTenant-AuthSource: MW3PR12MB4553.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 May 2024 19:12:36.6411
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: x+TYGg+zCu0ERJVtKa2ZPmwylYXhJWmy36CnJuVwWXqyg5HWcMiCatFC+5OgZ58d
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8562

Hi Reinette,

On 5/3/24 18:30, Reinette Chatre wrote:
> Hi Babu,
> 
> On 3/28/2024 6:06 PM, Babu Moger wrote:
>> Add the functionality to enable/disable ABMC feature.
>>
>> ABMC is enabled by setting enabled bit(0) in MSR L3_QOS_EXT_CFG. When the
>> state of ABMC is changed, it must be changed to the updated value on all
>> logical processors in the QOS Domain.
> 
> This patch does much more than enable what is mentioned above. There is little
> information about what this patch aims to accomplish. Without this it makes
> review difficult.

Sure. Also I need to add details about why resctrl_arch_reset_rmid_all()
is required. Will do.

> 
>>
>> The ABMC feature details are documented in APM listed below [1].
>> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
>> Publication # 24593 Revision 3.41 section 19.3.3.3 Assignable Bandwidth
>> Monitoring (ABMC).
>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
>> ---
>> v3: No changes.
>>
>> v2: Few text changes in commit message.
>> ---
>>  arch/x86/include/asm/msr-index.h       |  1 +
>>  arch/x86/kernel/cpu/resctrl/internal.h | 12 ++++
>>  arch/x86/kernel/cpu/resctrl/rdtgroup.c | 76 +++++++++++++++++++++++++-
>>  3 files changed, 88 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
>> index 05956bd8bacf..f16ee50b1a23 100644
>> --- a/arch/x86/include/asm/msr-index.h
>> +++ b/arch/x86/include/asm/msr-index.h
>> @@ -1165,6 +1165,7 @@
>>  #define MSR_IA32_MBA_BW_BASE		0xc0000200
>>  #define MSR_IA32_SMBA_BW_BASE		0xc0000280
>>  #define MSR_IA32_EVT_CFG_BASE		0xc0000400
>> +#define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
>>  
>>  /* MSR_IA32_VMX_MISC bits */
>>  #define MSR_IA32_VMX_MISC_INTEL_PT                 (1ULL << 14)
>> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
>> index 722388621403..8238ee437369 100644
>> --- a/arch/x86/kernel/cpu/resctrl/internal.h
>> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
>> @@ -96,6 +96,9 @@ cpumask_any_housekeeping(const struct cpumask *mask, int exclude_cpu)
>>  	return cpu;
>>  }
>>  
>> +/* ABMC ENABLE */
> 
> Can this comment be made more useful?

How about?
/* Setting bit 0 in L3_QOS_EXT_CFG enables ABMC features */

Or I can remove it totally.

> 
>> +#define ABMC_ENABLE			BIT(0)
>> +
>>  struct rdt_fs_context {
>>  	struct kernfs_fs_context	kfc;
>>  	bool				enable_cdpl2;
>> @@ -433,6 +436,7 @@ struct rdt_parse_data {
>>   * @mbm_cfg_mask:	Bandwidth sources that can be tracked when Bandwidth
>>   *			Monitoring Event Configuration (BMEC) is supported.
>>   * @cdp_enabled:	CDP state of this resource
>> + * @abmc_enabled:	ABMC feature is enabled
>>   *
>>   * Members of this structure are either private to the architecture
>>   * e.g. mbm_width, or accessed via helpers that provide abstraction. e.g.
>> @@ -448,6 +452,7 @@ struct rdt_hw_resource {
>>  	unsigned int		mbm_width;
>>  	unsigned int		mbm_cfg_mask;
>>  	bool			cdp_enabled;
>> +	bool			abmc_enabled;
>>  };
>>  
>>  static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *r)
>> @@ -491,6 +496,13 @@ static inline bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l)
>>  
>>  int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable);
>>  
>> +static inline bool resctrl_arch_get_abmc_enabled(enum resctrl_res_level l)
>> +{
>> +	return rdt_resources_all[l].abmc_enabled;
>> +}
>> +
>> +int resctrl_arch_set_abmc_enabled(enum resctrl_res_level l, bool enable);
>> +
>>  /*
>>   * To return the common struct rdt_resource, which is contained in struct
>>   * rdt_hw_resource, walk the resctrl member of struct rdt_hw_resource.
>> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> index 05f551bc316e..f49073c86884 100644
>> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> @@ -850,9 +850,15 @@ static int rdtgroup_mbm_assign_show(struct kernfs_open_file *of,
>>  				    struct seq_file *s, void *v)
>>  {
>>  	struct rdt_resource *r = of->kn->parent->priv;
>> +	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
>>  
>> -	if (r->mbm_assign_capable)
>> +	if (r->mbm_assign_capable && hw_res->abmc_enabled) {
>> +		seq_puts(s, "[abmc]\n");
>> +		seq_puts(s, "legacy_mbm\n");
>> +	} else if (r->mbm_assign_capable) {
>>  		seq_puts(s, "abmc\n");
>> +		seq_puts(s, "[legacy_mbm]\n");
>> +	}
>>  
>>  	return 0;
>>  }
>> @@ -2433,6 +2439,74 @@ int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable)
>>  	return 0;
>>  }
>>  
>> +static void resctrl_abmc_msrwrite(void *arg)
>> +{
>> +	bool *enable = arg;
>> +	u64 msrval;
>> +
>> +	rdmsrl(MSR_IA32_L3_QOS_EXT_CFG, msrval);
>> +
>> +	if (*enable)
>> +		msrval |= ABMC_ENABLE;
>> +	else
>> +		msrval &= ~ABMC_ENABLE;
>> +
>> +	wrmsrl(MSR_IA32_L3_QOS_EXT_CFG, msrval);
>> +}
>> +
>> +static int resctrl_abmc_setup(enum resctrl_res_level l, bool enable)
>> +{
>> +	struct rdt_resource *r = &rdt_resources_all[l].r_resctrl;
>> +	struct rdt_domain *d;
>> +
>> +	/* Update QOS_CFG MSR on all the CPUs in cpu_mask */
> 
> "all the CPUs in cpu_mask" -> "all the CPUs associated with the resource"?

Sure.

> 
>> +	list_for_each_entry(d, &r->domains, list) {
>> +		on_each_cpu_mask(&d->cpu_mask, resctrl_abmc_msrwrite, &enable, 1);
>> +		resctrl_arch_reset_rmid_all(r, d);
> 
> Could the changelog please explain why this is needed and what the impact of
> this is?

Sure.

> 
>> +	}
>> +
>> +	return 0;
>> +}
> 
> I think the naming can be changed to make these easier to understand. For example,
> resctrl_abmc_msrwrite() -> resctrl_abmc_set_one()
> resctrl_abmc_setup() -> resctrl_abmc_set_all()

Sure.

> 
>> +
>> +static int resctrl_abmc_enable(enum resctrl_res_level l)
>> +{
>> +	struct rdt_hw_resource *hw_res = &rdt_resources_all[l];
>> +	int ret = 0;
>> +
>> +	if (!hw_res->abmc_enabled) {
>> +		ret = resctrl_abmc_setup(l, true);
>> +		if (!ret)
>> +			hw_res->abmc_enabled = true;
>> +	}
>> +
>> +	return ret;
>> +}
>> +
>> +static void resctrl_abmc_disable(enum resctrl_res_level l)
>> +{
>> +	struct rdt_hw_resource *hw_res = &rdt_resources_all[l];
>> +
>> +	if (hw_res->abmc_enabled) {
>> +		resctrl_abmc_setup(l, false);
>> +		hw_res->abmc_enabled = false;
>> +	}
>> +}
>> +
>> +int resctrl_arch_set_abmc_enabled(enum resctrl_res_level l, bool enable)
>> +{
>> +	struct rdt_hw_resource *hw_res = &rdt_resources_all[l];
>> +
>> +	if (!hw_res->r_resctrl.mbm_assign_capable)
>> +		return -EINVAL;
>> +
>> +	if (enable)
>> +		return resctrl_abmc_enable(l);
>> +
>> +	resctrl_abmc_disable(l);
>> +
>> +	return 0;
>> +}
> 
> Why is resctrl_arch_set_abmc_enabled() necessary? It seem to add an unnecessary
> layer of abstraction.
> 

I feel it is better to keep it that way. It is consistent with definition
of resctrl_arch_set_cdp_enabled. It handles both enable and disable.
Otherwise we have add those checks from the caller.

-- 
Thanks
Babu Moger