Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4;
Message-ID: <2d736f74-c692-4e06-8e61-8a068e333db8@intel.com>
Date:   Mon, 30 Oct 2023 14:23:03 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v9 7/8] x86/resctrl: Sub NUMA Cluster detection and enable
Content-Language: en-US
To:     Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>,
        "Peter Newman" <peternewman@google.com>,
        Jonathan Corbet <corbet@lwn.net>,
        "Shuah Khan" <skhan@linuxfoundation.org>, <x86@kernel.org>
CC:     Shaopeng Tan <tan.shaopeng@fujitsu.com>,
        James Morse <james.morse@arm.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Babu Moger <babu.moger@amd.com>,
        Randy Dunlap <rdunlap@infradead.org>,
        <linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
        <patches@lists.linux.dev>
References: <20231003213043.13565-1-tony.luck@intel.com>
 <20231020213100.123598-1-tony.luck@intel.com>
 <20231020213100.123598-8-tony.luck@intel.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20231020213100.123598-8-tony.luck@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RGhGbFc2cnZUeTFkU0QwRTQ2SVkvdHJqNWVWSFdmdml4TXNpR0htdWlvRWpp?=
 =?utf-8?B?TFhMcVMyZFNnLzZlQk1zVE03M0hiUzg3RytPaVVxOGs0a1RkRGo1aU13eTdw?=
 =?utf-8?B?SU5tVExTbDNhdTFjRG9KSzl3WWZNS3k0enBweUtEMGNsL0l6U1FlK2JEWXRI?=
 =?utf-8?B?RDIzYjJzYkZ0RlgycWNEaUlJZ25PSDkySUxoYUxneGh2SU0wNkR4Nm1CUmt1?=
 =?utf-8?B?NldXd0g0RWhyL0w3bkk5TUZ6RzRTZWVIUGlFRnRxS2gvS0tMQk9xYzFTeENW?=
 =?utf-8?B?R3BvTm5RVzYyMFRobHYrRm9oZ0F1aW1CdnNGV3N3NkxmSjkzQWdxck92VUxU?=
 =?utf-8?B?NTU5L1I4UE5LSXVuejhGL3EwRU5xQSszWGlVTW1hTHFVM0RTbVdHbThqczZq?=
 =?utf-8?B?OE1pQXNScmRIUkpTdlpxaVYyZk9qekNGT2s5S0QwVmh6MG9DQnJtVW80UXVB?=
 =?utf-8?B?SmxOWXdGSVVnOWNQemQwWFZ1NGxiRU9kVWxjazlya2crZmRGZlphQlBHZm10?=
 =?utf-8?B?bzV2OFUzTGJnc2ZkU2RPMXlGbEVUZ2JlMWJDZ0NzcUIyOGQzSEtKTlJ3RytI?=
 =?utf-8?B?MXE1MXFRenFxZU13NlpkS2wwM1JXN2ZGZUo3Tzh6SmF2SWdKVHVvZjdRall0?=
 =?utf-8?B?U3A0dzRDMzZ4ZTA2ZE9Xa0NkTGsxT0dzQjFyaVFDZlhTZmxPaHBKaWUyRmI1?=
 =?utf-8?B?ckRMT3dNRm1oTW9IYjVpZjcvaFpTQ0FlRDdkRnNzM3BXM1BXanBhRitjT0dx?=
 =?utf-8?B?bUI4TForeitmcVM3Njc5azhFeS82ZitIYzR1VGxqKzBLTGM4Q2FnTUIxY0Zn?=
 =?utf-8?B?YU16WVl1WjhyUUhmOUJyUTdMUDZFcjBLTG5NR1B3OXFST3JMQzNmK0JDRlhW?=
 =?utf-8?B?cDVaMEV2NzhQWXJjcHhJZXlsTzc0UCtoNEJxMDhBS1B0S0VTYndJTXhLblVT?=
 =?utf-8?B?TmMvWHh3T0w5TWdvZTVPN3JWZ093K0Y3bHdTaWwyb2QxNGtiVVNRRGQyc1pt?=
 =?utf-8?B?T1ZBYTZiVjNKbDdzMFJTSVBsWGVRVG9qb3ZXYXpMcjRTMnFUeXk4b08rWTJk?=
 =?utf-8?B?bUhjWFYvdCswR2ZSK0FKM01TWUZGZTAwQlNhZ2lPLzdHQm1vekZTbmkxZlg3?=
 =?utf-8?B?WXV4SUZmdmxjQmFIYmpNaFhqUHVUSGhTbmtHeW9UcCswVEkxYkFkRWE3RXlr?=
 =?utf-8?B?YXpSSVNkeVVITjRWaFJ4KzEwSzlSRWZDdzU0L0ErekhvbUVvM1pPSVlmRUxw?=
 =?utf-8?B?bmpza2lMd09jclVuS0lGT0E3YVRYS050UlFLM0ExeXdxQ0VJYXJ0WkVLL3h3?=
 =?utf-8?B?ZnFjNlBSTnNWTHpMQ2owc3N3NkhQMC9DK095Y0JEZ1BnQk04VFd5bTlsOTZy?=
 =?utf-8?B?RnVHR0JpMzFrVUFUalU3MElqOU5YdXZUZGFsTURGSTBDMWdzcGhtdzFKMXRP?=
 =?utf-8?B?eEpoZzAyaGN0UmFObjkyVlNSOGkrWTNkMmFEWVIrOVJLTXFUT3MyWi9kYWdx?=
 =?utf-8?B?dWpHZlRuelMrV1dhaTNmZE9kRlZnNG9wRnR5SUFMR1haL1BhbjFDZ2ZUVEpu?=
 =?utf-8?B?RnVYQWNUa0xvdFM1NFA5YktWeFE2UzBBSkVYWjhFMzdCanhCc2VJUFUvUjhY?=
 =?utf-8?B?Q1NmTDY4RlljYTgrYjd0T2JQYWdqUFlkMnUySkRhdEx5NHZiYzVZY0RocWVE?=
 =?utf-8?B?SWJMaE52QmpFU1dOSjcvUWR2dStFZVdzbGhjS2dvZTNRN2o1ZWZoZG5QZGQx?=
 =?utf-8?B?dWZCYXZOWmU1WnlGNzdDZU0xU1BRTkZYRE41WXBzVGl3dlRYY1BKSnNOM0Yr?=
 =?utf-8?B?K3E0NE15VFJqdjJCZEw2c25HWDkrVCtmYzZBQUNKN3VXc0JCcUhzTThwQm5y?=
 =?utf-8?B?ajlwelpPMW4rOTFXekVkczdTcFJMWU95a0tRNHZBVHBrQk5BRHRpZENFRjNQ?=
 =?utf-8?B?VlBYa3ZTR25sQzB1N0NVN3J6OTdxY2k0T0RyK2N2cGhTRjRnSHNHMjlxL3A1?=
 =?utf-8?B?LzJaeVZpTWptdXFPQU95aG94eWV5VVprZnAreGpqbHJwMkUvUGw3aDZZN2xS?=
 =?utf-8?B?L0xqbHJrdGRQaWxrWURaaUh6d2hwaExvNEh2cFJvOVBjSVNoQmhOWE1BOUZm?=
 =?utf-8?B?K3BYRzIwUlZ3cnRvb1JJdklpVkY5ZHdhQzBVYmhaQkhIRk5iWDk3ckVINm5H?=
 =?utf-8?B?S0E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a195bc8e-4230-4e6d-b77c-08dbd98e6773
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Oct 2023 21:23:04.8438
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 0+Km+dc+c3o1ipc3K5WV8ODEIRAR5TkZ32HU9N0ZE2GS2v/i3xYP9vNO5/VR5D/1QkmNpA5/l2DuT8o9w8ry7FOXFjbjTnKzM3yi7B52/0M=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR11MB7921
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-1.3 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 30 Oct 2023 14:23:22 -0700 (PDT)

Hi Tony,

On 10/20/2023 2:30 PM, Tony Luck wrote:
> There isn't a simple h/w bit that indicates whether a CPU is

"h/w" -> hardware

> running in Sub NUMA Cluster (SNC) mode. Infer the state by comparing
> the ratio of NUMA nodes to L3 cache instances.
> 
> When SNC mode is detected, reconfigure the RMID counters by updating
> the MSR_RMID_SNC_CONFIG MSR on each socket as CPUs are seen.
> 
> Clearing bit zero of the MSR divides the RMIDs and renumbers the ones
> on the second SNC node to start from zero. An earlier commit includes

Please drop the "earlier commit" reference.

> all the required changes in Linux to operate in this reconfigured mode.
> 
> Reviewed-by: Peter Newman <peternewman@google.com>
> Signed-off-by: Tony Luck <tony.luck@intel.com>
> ---
> Changes since v6:
> 
> Moved kfree(node_caches); earlier, to the earliest point where it
> is no longer needed.
> 
> Added Granite Rapids to list of CPU models that support SNC mode.
> 
> Added Peter's review tag
> 
>  arch/x86/include/asm/msr-index.h   |  1 +
>  arch/x86/kernel/cpu/resctrl/core.c | 92 ++++++++++++++++++++++++++++++
>  2 files changed, 93 insertions(+)
> 
> diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
> index e3fa9cecd599..4285a5ee81fe 100644
> --- a/arch/x86/include/asm/msr-index.h
> +++ b/arch/x86/include/asm/msr-index.h
> @@ -1109,6 +1109,7 @@
>  #define MSR_IA32_QM_CTR			0xc8e
>  #define MSR_IA32_PQR_ASSOC		0xc8f
>  #define MSR_IA32_L3_CBM_BASE		0xc90
> +#define MSR_RMID_SNC_CONFIG		0xca0
>  #define MSR_IA32_L2_CBM_BASE		0xd10
>  #define MSR_IA32_MBA_THRTL_BASE		0xd50
>  
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 0e418dd14070..ac187eb0440f 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -16,11 +16,14 @@
>  
>  #define pr_fmt(fmt)	"resctrl: " fmt
>  
> +#include <linux/cpu.h>
>  #include <linux/slab.h>
>  #include <linux/err.h>
>  #include <linux/cacheinfo.h>
>  #include <linux/cpuhotplug.h>
> +#include <linux/mod_devicetable.h>
>  
> +#include <asm/cpu_device_id.h>
>  #include <asm/intel-family.h>
>  #include <asm/resctrl.h>
>  #include "internal.h"
> @@ -755,11 +758,42 @@ static void clear_closid_rmid(int cpu)
>  	wrmsr(MSR_IA32_PQR_ASSOC, 0, 0);
>  }
>  
> +/*
> + * The power-on reset value of MSR_RMID_SNC_CONFIG is 0x1
> + * which indicates that RMIDs are configured in legacy mode.
> + * This mode is incompatible with Linux resctrl semantics
> + * as RMIDs are partitioned between SNC nodes, which requires
> + * a user to know which RMID is allocated to a task.
> + * Clearing bit 0 reconfigures the RMID counters for use
> + * in Sub NUMA Cluster mode. This mode is better for Linux.
> + * The RMID space is divided between all SNC nodes with the
> + * RMIDs renumbered to start from zero in each node when
> + * couning operations from tasks. Code to read the counters
> + * must adjust RMID counnter numbers based on SNC node. See

counnter -> counter

> + * __rmid_read() for code that does this.
> + */
> +static void snc_remap_rmids(int cpu)
> +{
> +	u64 val;
> +
> +	/* Only need to enable once per package. */
> +	if (cpumask_first(topology_core_cpumask(cpu)) != cpu)
> +		return;
> +
> +	rdmsrl(MSR_RMID_SNC_CONFIG, val);
> +	val &= ~BIT_ULL(0);
> +	wrmsrl(MSR_RMID_SNC_CONFIG, val);
> +}
> +
>  static int resctrl_online_cpu(unsigned int cpu)
>  {
>  	struct rdt_resource *r;
>  
>  	mutex_lock(&rdtgroup_mutex);
> +
> +	if (snc_nodes_per_l3_cache > 1)
> +		snc_remap_rmids(cpu);
> +
>  	for_each_capable_rdt_resource(r)
>  		domain_add_cpu(cpu, r);
>  	/* The cpu is set in default rdtgroup after online. */
> @@ -1014,11 +1048,69 @@ static __init bool get_rdt_resources(void)
>  	return (rdt_mon_capable || rdt_alloc_capable);
>  }
>  
> +/* CPU models that support MSR_RMID_SNC_CONFIG */
> +static const struct x86_cpu_id snc_cpu_ids[] __initconst = {
> +	X86_MATCH_INTEL_FAM6_MODEL(ICELAKE_X, 0),
> +	X86_MATCH_INTEL_FAM6_MODEL(SAPPHIRERAPIDS_X, 0),
> +	X86_MATCH_INTEL_FAM6_MODEL(EMERALDRAPIDS_X, 0),
> +	X86_MATCH_INTEL_FAM6_MODEL(GRANITERAPIDS_X, 0),
> +	{}
> +};
> +
> +/*
> + * There isn't a simple h/w bit that indicates whether a CPU is running

h/w -> hardware

> + * in Sub NUMA Cluster (SNC) mode. Infer the state by comparing the
> + * ratio of NUMA nodes to L3 cache instances.
> + * It is not possible to accurately determine SNC state if the system is
> + * booted with a maxcpus=N parameter. That distorts the ratio of SNC nodes

Can the user be warned when SNC state cannot be determined accurately? 

> + * to L3 caches. It will be OK if system is booted with hyperthreading
> + * disabled (since this doesn't affect the ratio).
> + */
> +static __init int snc_get_config(void)
> +{
> +	unsigned long *node_caches;
> +	int mem_only_nodes = 0;
> +	int cpu, node, ret;
> +	int num_l3_caches;
> +
> +	if (!x86_match_cpu(snc_cpu_ids))
> +		return 1;
> +
> +	node_caches = bitmap_zalloc(nr_node_ids, GFP_KERNEL);
> +	if (!node_caches)
> +		return 1;
> +
> +	cpus_read_lock();
> +	for_each_node(node) {
> +		cpu = cpumask_first(cpumask_of_node(node));
> +		if (cpu < nr_cpu_ids)
> +			set_bit(get_cpu_cacheinfo_id(cpu, 3), node_caches);
> +		else
> +			mem_only_nodes++;
> +	}
> +	cpus_read_unlock();
> +
> +	num_l3_caches = bitmap_weight(node_caches, nr_node_ids);
> +	kfree(node_caches);
> +
> +	if (!num_l3_caches)
> +		return 1;
> +
> +	ret = (nr_node_ids - mem_only_nodes) / num_l3_caches;
> +
> +	if (ret > 1)
> +		rdt_resources_all[RDT_RESOURCE_L3].r_resctrl.mon_scope = RESCTRL_NODE;
> +
> +	return ret;
> +}
> +
>  static __init void rdt_init_res_defs_intel(void)
>  {
>  	struct rdt_hw_resource *hw_res;
>  	struct rdt_resource *r;
>  
> +	snc_nodes_per_l3_cache = snc_get_config();
> +
>  	for_each_rdt_resource(r) {
>  		hw_res = resctrl_to_arch_res(r);
>  

Reinette