Received-SPF: pass (google.com: domain of linux-kernel+bounces-186734-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <e7ccc2f0-c2aa-4814-a2be-5a5a90542073@intel.com>
Date: Wed, 22 May 2024 14:26:48 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v18 16/17] x86/resctrl: Sub NUMA Cluster detection and
 enable
To: Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>, Maciej
 Wieczor-Retman <maciej.wieczor-retman@intel.com>, Peter Newman
	<peternewman@google.com>, James Morse <james.morse@arm.com>, Babu Moger
	<babu.moger@amd.com>, Drew Fustini <dfustini@baylibre.com>, Dave Martin
	<Dave.Martin@arm.com>
CC: <x86@kernel.org>, <linux-kernel@vger.kernel.org>,
	<patches@lists.linux.dev>
References: <20240515222326.74166-1-tony.luck@intel.com>
 <20240515222326.74166-17-tony.luck@intel.com>
Content-Language: en-US
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20240515222326.74166-17-tony.luck@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?N2xYSUJ6Q1hwQTFmSXg4Q1lYR0dqdzRjMXVrS25Ha01lMEVkMlRkVllDemRh?=
 =?utf-8?B?TGt0NUFvcWx3L0pDVGlJNFYzR1NuREg2blcvUXR0akFJazJuK2F0QUp0c3Bu?=
 =?utf-8?B?WUlPOTZJQzJITmNmZmxsMFZMV3BZbHR3SFlZSVdkLzdPTjg1aEtxcmhHR3cw?=
 =?utf-8?B?OXhQWERXRmtVR2p0cHg2U0NOMGV6NmE3a29jeVVOT0gwOWlieVNkYUdQR3c5?=
 =?utf-8?B?Sk5TcUJ6WjBDZmxUSmRyazlEMWJlV0hoeHBadnNnZVRqVDJCdml4T3Ztc2Fk?=
 =?utf-8?B?OEk0SkovaE8xakJqa3ZHQlBHNkd3bHVXVU1TaFZYRHlBKzU5dVdWMDRrVlNu?=
 =?utf-8?B?YU5jN04zTm8vRzBJMGRZVkpqbDdXaDY1bzZDNSt4NkcxdWRFbWE1TVluLzJi?=
 =?utf-8?B?UDhvSEJaR1g5K3pEbjR0eTQwdmJDTWxZZjlqbzBNSm5qU0FEU3V5V0w1ZC9y?=
 =?utf-8?B?aVd1anZZMHUxNC95TXlRNXBnSHB4VmpYUlJHU1hMbWVtdDk3bnBGMzZVaEh5?=
 =?utf-8?B?cGJhTkJuVlJIcm9SVGhBRXlUM29EU2VnZ1JmM1dWZnhEeTI2c2VwV0hNTGUz?=
 =?utf-8?B?Z2dTN2M5dlJxV1pNMEwwdUUxSWFabWVsZDZtZE15azl3Mks0N2hmOGo2ZTEv?=
 =?utf-8?B?UmQ1ZFpsSCsxbjg5OFlGSDN1QlVUaW5kL2hJNEZsYWhWVFBtTytzVGRYMyt2?=
 =?utf-8?B?SGNDTzI0aDFCUTQwdFQzN29WSkNPU2ZOTlV1Q1ZPbDREcDRFY3NaeVdkQitD?=
 =?utf-8?B?bnUyZk1xWTloaXN5K3VwWk9WRlR3MTRnQmp3dlQ4NFFsMFI0Y1Flb05PVlhP?=
 =?utf-8?B?QXFPeEVxTUdJZDB6cXpyWGU0dzVoV3E0WVFyOXlVbjloclJJMTJGTkVFVmZz?=
 =?utf-8?B?UTlaMWJsbG5aV0NlaW10c25GczIrV0k4dmVNa0xXWHc0M2g3OGlBcjB0V0tl?=
 =?utf-8?B?ZkZqVkVsaUwvd3doMnZlbE8yZWxYQ0Y0cGlXRTl4RDhkTG5zYmRpY3cwUWFj?=
 =?utf-8?B?OGgxck83WmpKQkFOSFJGL1lLT0VyR2d6V2IzZjRwdEtOOUlpSkkweCtSRVh4?=
 =?utf-8?B?aCtuZVd3QlEzUWZGVTcrdFNSTlV4Y3JFS3pKMEd3bVQ3bm5kUEtwTE9aRFRr?=
 =?utf-8?B?SWp1KzFSdjdBR3NORXRGOSsvdUlXVndHdE5oQjcxL2d0UTNKSHR5UTYycGh6?=
 =?utf-8?B?TnpzcUdTT1lGK2FpOTMyenNKd25halNGSmVna2xyamJxcU9LTUxzcHpHRm5v?=
 =?utf-8?B?NEV3WFN2NTdXWjc2bm5YM1F5aFJlRmhsdjZzdkdqK0lFRGZSazVtOXRaSEtH?=
 =?utf-8?B?a1UxNVoxeWFCZUc2cjRmMVpzRFUyeVV6dXd1V2l5c25tS0tNRlBVZEM2WEkz?=
 =?utf-8?B?aUxMcTVoRWx4cWNKOXVBdGxaYVZFMERyTmkybndoUTNqL3NsYzZLcU5uandT?=
 =?utf-8?B?YUVibENOUkc4Snl3WGRYNklBbWRJK2tiN0dGSmtETkE5WEtvRGtDS3o1aVpx?=
 =?utf-8?B?VFZqdjVsK1RZNWZhNDd6ZmQxclBwWG9ONzJpd0tOUm5acTlQWm52WjhpdkdE?=
 =?utf-8?B?TUkvUXVjY2hBN2EzM0kySThnVWlwYjVOZjJ1SEphOTlCN01JT2tGemNvN0Vr?=
 =?utf-8?B?ZWtWL2t5VHNEcktKOWtFdEJoeTBwMUlYNTZnbFg5d09jSkdQMzdpMzUza2M1?=
 =?utf-8?B?WWhJU0txcXg5Q2VkMjR4MjVjRDRDeEhzc3ZmUDE4V3lOV1lsRjNRb1pzWTNJ?=
 =?utf-8?B?M1pPa2lqSTR2UUh3bndoQ2VYdHlGNzQzaCtXWHJpRitFaU1rWElYWWdaMkpV?=
 =?utf-8?B?NnN2R2tVenMyVkhIVXB5UDE3YVFVMnpQTURtVTVlYnpvZ3o5ZHNIMzkwdFRL?=
 =?utf-8?B?clFtYkJ1ZS9pdWFoU2hVTm9vTDNCZzI3MHZHS3ZwdS9pRUxFRVRJM3g0OTVk?=
 =?utf-8?B?Nk1CczlpQ3Z3R0xtVlRWUVc3aVBxYWZDUU5ZbjFGTmJKZkVrRkJTS05tRXZY?=
 =?utf-8?B?S1lhcDdQeTVKdzAwQlZ0UjN3bVg5dGZVcTlVRUpXd1JLN3NjKy9KdWl5R2Jj?=
 =?utf-8?B?enp4TlVDanVxeWoyZGZhdzRZVG9UUXpJRDZJdWtrWHFneFRWUS81ZDMyWXdw?=
 =?utf-8?B?WkNHRC9mRjBrM1FEUVVoMGZhTm16VGZsRGVxdkY0NnFXOXlROEZhMkRMK1FM?=
 =?utf-8?B?ZkE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ec56cb61-3c30-478a-a753-08dc7aa5e4cb
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 May 2024 21:26:50.7419
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: u0yPjOoyRfF7/qpLVgGuACA6O5zj9iPkS/XNjW+hoccK3J1KfUtqok/Uhl/UYRt4WW+Md5IiqHXsQX4DdDCkoUpkAHLpypS652su2XAr4PY=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB7422
X-OriginatorOrg: intel.com

Hi Tony,

On 5/15/2024 3:23 PM, Tony Luck wrote:
> There isn't a simple hardware bit that indicates whether a CPU is
> running in Sub NUMA Cluster (SNC) mode. Infer the state by comparing

Sometimes it is "Sub-NUMA Cluster" and sometimes it is "Sub NUMA Cluster".

> number CPUs sharing the L3 cache with CPU0 to the number of CPUs in
> the same NUMA node as CPU0.
> 
> When SNC mode is detected, reconfigure the RMID counters by updating
> the MSR_RMID_SNC_CONFIG MSR on each socket as CPUs are seen.
> 
> Clearing bit zero of the MSR divides the RMIDs and renumbers the ones
> on the second SNC node to start from zero.
> 
> Signed-off-by: Tony Luck <tony.luck@intel.com>
> ---
>  arch/x86/include/asm/msr-index.h   |   1 +
>  arch/x86/kernel/cpu/resctrl/core.c | 104 +++++++++++++++++++++++++++++
>  2 files changed, 105 insertions(+)
> 
> diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
> index e022e6eb766c..3cb8dd6311c3 100644
> --- a/arch/x86/include/asm/msr-index.h
> +++ b/arch/x86/include/asm/msr-index.h
> @@ -1164,6 +1164,7 @@
>  #define MSR_IA32_QM_CTR			0xc8e
>  #define MSR_IA32_PQR_ASSOC		0xc8f
>  #define MSR_IA32_L3_CBM_BASE		0xc90
> +#define MSR_RMID_SNC_CONFIG		0xca0
>  #define MSR_IA32_L2_CBM_BASE		0xd10
>  #define MSR_IA32_MBA_THRTL_BASE		0xd50
>  
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index dd40c998df72..195f9e29c553 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -21,6 +21,7 @@
>  #include <linux/err.h>
>  #include <linux/cacheinfo.h>
>  #include <linux/cpuhotplug.h>
> +#include <linux/mod_devicetable.h>
>  
>  #include <asm/cpu_device_id.h>
>  #include <asm/resctrl.h>
> @@ -753,11 +754,42 @@ static void clear_closid_rmid(int cpu)
>  	      RESCTRL_RESERVED_CLOSID);
>  }
>  
> +/*
> + * The power-on reset value of MSR_RMID_SNC_CONFIG is 0x1
> + * which indicates that RMIDs are configured in legacy mode.
> + * This mode is incompatible with Linux resctrl semantics
> + * as RMIDs are partitioned between SNC nodes, which requires
> + * a user to know which RMID is allocated to a task.
> + * Clearing bit 0 reconfigures the RMID counters for use
> + * in Sub NUMA Cluster mode. This mode is better for Linux.
> + * The RMID space is divided between all SNC nodes with the
> + * RMIDs renumbered to start from zero in each node when
> + * couning operations from tasks. Code to read the counters

couning -> counting

> + * must adjust RMID counter numbers based on SNC node. See
> + * __rmid_read() for code that does this.
> + */
> +static void snc_remap_rmids(int cpu)
> +{
> +	u64 val;
> +
> +	/* Only need to enable once per package. */
> +	if (cpumask_first(topology_core_cpumask(cpu)) != cpu)
> +		return;
> +
> +	rdmsrl(MSR_RMID_SNC_CONFIG, val);
> +	val &= ~BIT_ULL(0);
> +	wrmsrl(MSR_RMID_SNC_CONFIG, val);
> +}
> +
>  static int resctrl_arch_online_cpu(unsigned int cpu)
>  {
>  	struct rdt_resource *r;
>  
>  	mutex_lock(&domain_list_lock);
> +
> +	if (snc_nodes_per_l3_cache > 1)
> +		snc_remap_rmids(cpu);
> +
>  	for_each_capable_rdt_resource(r)
>  		domain_add_cpu(cpu, r);
>  	mutex_unlock(&domain_list_lock);
> @@ -997,11 +1029,83 @@ static __init bool get_rdt_resources(void)
>  	return (rdt_mon_capable || rdt_alloc_capable);
>  }
>  
> +/* CPU models that support MSR_RMID_SNC_CONFIG */
> +static const struct x86_cpu_id snc_cpu_ids[] __initconst = {
> +	X86_MATCH_VFM(INTEL_ICELAKE_X, 0),
> +	X86_MATCH_VFM(INTEL_SAPPHIRERAPIDS_X, 0),
> +	X86_MATCH_VFM(INTEL_EMERALDRAPIDS_X, 0),
> +	X86_MATCH_VFM(INTEL_GRANITERAPIDS_X, 0),
> +	X86_MATCH_VFM(INTEL_ATOM_CRESTMONT_X, 0),
> +	{}
> +};
> +
> +/*
> + * There isn't a simple hardware bit that indicates whether a CPU is running
> + * in Sub NUMA Cluster (SNC) mode. Infer the state by comparing the
> + * number CPUs sharing the L3 cache with CPU0 to the number of CPUs in
> + * the same NUMA node as CPU0.
> + * It is not possible to accurately determine SNC state if the system is
> + * booted with a maxcpus=N parameter. That distorts the ratio of SNC nodes
> + * to L3 caches. It will be OK if system is booted with hyperthreading
> + * disabled (since this doesn't affect the ratio).
> + */
> +static __init int snc_get_config(void)
> +{
> +	struct cpu_cacheinfo *ci = get_cpu_cacheinfo(0);
> +	const cpumask_t	*node0_cpumask;
> +	cpumask_t *l3_cpumask = NULL;
> +	int i, ret;
> +
> +	if (!x86_match_cpu(snc_cpu_ids))
> +		return 1;
> +
> +	cpus_read_lock();
> +	if (num_online_cpus() != num_present_cpus())
> +		pr_warn("Some CPUs offline, SNC detection may be incorrect\n");
> +	cpus_read_unlock();
> +
> +	for (i = 0; i < ci->num_leaves; i++) {
> +		if (ci->info_list[i].level == 3) {
> +			if (ci->info_list[i].attributes & CACHE_ID) {
> +				l3_cpumask = &ci->info_list[i].shared_cpu_map;
> +				break;
> +			}
> +		}
> +	}
> +	if (!l3_cpumask) {
> +		pr_info("can't get CPU0 L3 mask\n");

Sentence can start with upper case

> +		return 1;
> +	}
> +
> +	node0_cpumask = cpumask_of_node(cpu_to_node(0));
> +
> +	ret = bitmap_weight(cpumask_bits(l3_cpumask), nr_cpu_ids) /
> +	       bitmap_weight(cpumask_bits(node0_cpumask), nr_cpu_ids);
> +

Can cpumask_weight() be used?

> +	/* sanity check: Only valid results are 1, 2, 3, 4 */
> +	switch (ret) {
> +	case 1:
> +		break;
> +	case 2 ... 4:
> +		pr_info("Sub-NUMA cluster detected with %d nodes per L3 cache\n", ret);
> +		rdt_resources_all[RDT_RESOURCE_L3].r_resctrl.mon_scope = RESCTRL_NODE;
> +		break;
> +	default:
> +		pr_warn("Ignore improbable SNC node count %d\n", ret);
> +		ret = 1;
> +		break;
> +	}
> +
> +	return ret;
> +}
> +
>  static __init void rdt_init_res_defs_intel(void)
>  {
>  	struct rdt_hw_resource *hw_res;
>  	struct rdt_resource *r;
>  
> +	snc_nodes_per_l3_cache = snc_get_config();
> +
>  	for_each_rdt_resource(r) {
>  		hw_res = resctrl_to_arch_res(r);
>  

Reinette