Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32;
Message-ID: <20c243fd-5068-b8a0-31d3-369b739ff1ca@intel.com>
Date:   Mon, 25 Sep 2023 16:29:44 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.15.1
Subject: Re: [PATCH v5 6/8] x86/resctrl: Sub NUMA Cluster detection and enable
Content-Language: en-US
To:     Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>,
        "Peter Newman" <peternewman@google.com>,
        Jonathan Corbet <corbet@lwn.net>,
        Shuah Khan <skhan@linuxfoundation.org>, <x86@kernel.org>
CC:     Shaopeng Tan <tan.shaopeng@fujitsu.com>,
        James Morse <james.morse@arm.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Babu Moger <babu.moger@amd.com>,
        Randy Dunlap <rdunlap@infradead.org>,
        <linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
        <patches@lists.linux.dev>
References: <20230722190740.326190-1-tony.luck@intel.com>
 <20230829234426.64421-1-tony.luck@intel.com>
 <20230829234426.64421-7-tony.luck@intel.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20230829234426.64421-7-tony.luck@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dUVBcWxWY09BQi9WSXlsTUNCZW0yTTVtejg1c2Q4OGF1RDV3MjlCYUtyeHpK?=
 =?utf-8?B?YWdKUGx1aWRJZDZCeGF5OFFaenpSTkJGWUJLOTBKQllQdFYwN3RSRTdxWHpM?=
 =?utf-8?B?UVFyTGlXeVExckdDOWd5OWdzVXhJQ3NSUUhPOHRBU2JBOGpPQ1h6MjlMb1ZD?=
 =?utf-8?B?aytMMityczhTTTNzL0JndjUyT3ZRYkcvVDhmc2tsZlFsTU9ybFN0RHBGakh3?=
 =?utf-8?B?Z1BGQWZhT0RFaGZTYUV0UTZncEMzWFZUTGlGUFhkMDhvSDE0bk1ZZFBTcFIr?=
 =?utf-8?B?ZWZLUjhaaHhxVTZOR01vK2E4bGR4S084ZUVQdWw5MDZoYktvSXhPTUt1UzhO?=
 =?utf-8?B?dmhWRDZtUmJDbmFrL05rZTlVS2lrMzBDZE05ZWNoYUpZWE9aU0k4UFVHU1pE?=
 =?utf-8?B?L3FPTmVpYVAwVlRnUmJOM3JMb0JSdXp1Q2RHU0xzMDhVSVFFUXRVSXZIRFlG?=
 =?utf-8?B?b2VwVlhJZlBIOEcrU1ErejVtaXdXSEk0Z1dWakQ5VWNsTW9WQVdDSnYxYUdP?=
 =?utf-8?B?d0w0Q0RhbXJ5bGUzN1RKZUN4U3BuSVEzTjBlbVFkQW1CUzl5Z253ZU92OFV3?=
 =?utf-8?B?eEhGU2RWLzd5UjNncVhIN3p0NUY2Y0kwTXZVYWVPdWFxRis3SU52RVFuWncx?=
 =?utf-8?B?YzZlOXo5eWpNa21BZ3dMMkQ2VklHZUlLRFpUcmdKYWtWUitlOGFBUEZqSkpz?=
 =?utf-8?B?VFZxSnFWTFNnWFRDekQ2cmk3d01KcHhtUmt6RFF4UGwwV0FKOXRDL3Qwanhr?=
 =?utf-8?B?NlpDaGRGTnIwS3B1NDFVM1J1dUhPdkRtcGdNRTlnRUlwd1pHRE9VQXQ3WWZm?=
 =?utf-8?B?Mms5emRuQjR3MHVTMldQNG00c1J3STc1d0RjUklBUUxOTTdJcGNxbmV1bmo4?=
 =?utf-8?B?aUdMR2lkTFNxdDBSRFNoWTMrZ2ZNS05HQmNKZE5NUVFpSlpUck9tM0Y3bEFn?=
 =?utf-8?B?ajlOc2VFb1BiYWozTFc5clRGOUVkTjZzNjBEWjg4dnNLWFd5bFNjU0N4RE4x?=
 =?utf-8?B?WTQzSGZzSWhCbXlMbFczMHVQTnhTVjVCTUwzc1RQdGVNb1N1RmVYZERCSWZw?=
 =?utf-8?B?bHN5SkE2aVB4cTRGOWFjTUg1eVUwRzBxa1o3ejlNNnlwY2k1L0FjR25YUnh0?=
 =?utf-8?B?LzBnYTJCV3ZxWjNOcjhFTGM0MFFmYzJCT3EyTUhFWGp4NVBtN09qbUY2SmQw?=
 =?utf-8?B?RThmaVFOcHZnNTEyRGpmdUpPalUzNEpPaXVEclFtL2VtR3o2NWpIUE1mSmgr?=
 =?utf-8?B?YU8xcVp4WVV0Z2RWUHhKSDU0S0lNTkkxNVI3M011MVZTd1l4STdUeklnbHVE?=
 =?utf-8?B?NnA5TTI1WC9Bd0VBUzNyaUtIdW9ob2llRUZ6Q0hvSkhpeEVtblB1eDJabDJF?=
 =?utf-8?B?ZENOOVdYVG5lU2w1RThWbVczQ3BBVEhaMzVUeUh0THZSTHY1NnY1bDNheDJr?=
 =?utf-8?B?VEx2emxPNUV3c3AwdWNMTE1NM0JMa1FjemxuQ25kUHFyQlFTUGRFRDhLVEpq?=
 =?utf-8?B?WWUwRFFqOTRlZyt4dUxvL21TcVNZSThRblFUcHcrQklKWFh1OGV3ZWVCV2lM?=
 =?utf-8?B?V0R5eGV2Vmx5OFRzQWVMNlpEYkg1WkZDZWZPVVZMOHp3aGRtSyt5MXoyNktp?=
 =?utf-8?B?aklPRHZKZVFKeVpjbUZ0NGFKWWdOd1B4RjBDL2VQTWNQOG9KdVlVeDB1VkVw?=
 =?utf-8?B?SnppQlpBQml6YnRseTQzeGJMaGQ5eUF4R0ZUdUtKdEE1YWdJUWp6YkNpbDRp?=
 =?utf-8?B?a21tL0tnamJxYWtCQkcva25wU3N2cnlVMENxUGJ2VytlZ3FpaHA3QmN1RnZH?=
 =?utf-8?B?R0JYOHNyRHZ3SUhDbE1UMGMxdU9tcGJlMUZieVRuV3U0MEtaQVVNV0orRGNz?=
 =?utf-8?B?VUwrTFpDWldEUlUwb2ZqbXVuOHlQRzdjeW5pRndSSDdBbWdsQm40Z1RkR1pw?=
 =?utf-8?B?RVRwVmlJeWdYQ0xLZGttWEtiRzI2WnBlSk5FVE9qVHVsRnpxOXE3ckJtTlh5?=
 =?utf-8?B?a1pRSmpVK2d0MHNTQzhwbnVCenlkMFUwYTRMYVBJajFmTG9jMGRkZU1BQzEv?=
 =?utf-8?B?YVNTM3BPeUcyaG5maGI1MFhpUCtEZE1XaE9YZTZmcjVDTzQ3NFRQcGZ4bk83?=
 =?utf-8?B?SnlSU3BBcllHV0lFekpzS1NTMzRLaXRRS3JSTGFZT3IyUXJJMkpxSkpPS1cw?=
 =?utf-8?B?N2c9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 3799f341-892c-4632-47ba-08dbbe1f4e78
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Sep 2023 23:29:47.4256
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: zrxrZQeTFa0pefmtIbU5zPxWLTamzc2B6Xege7S9FFwIfSg9ZJIH4XeV7mxpk5GolaaVExn7OHEV0zGTcB5+uNBwOMPQeZpphLN1uSSssBg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7683
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.3 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 25 Sep 2023 16:30:00 -0700 (PDT)

Hi Tony,

On 8/29/2023 4:44 PM, Tony Luck wrote:
> There isn't a simple h/w bit that indicates whether a CPU is
> running in Sub NUMA Cluster mode. Infer the state by comparing

Sub NUMA Cluster (SNC)

> the ratio of NUMA nodes to L3 cache instances.
> 
> When SNC mode is detected, reconfigure the RMID counters by updating
> the MSR_RMID_SNC_CONFIG MSR on each socket as CPUs are seen.

Can it be explained how RMID counters are reconfigured when
the MSR is updated?

> 
> Signed-off-by: Tony Luck <tony.luck@intel.com>
> ---
>  arch/x86/include/asm/msr-index.h   |  1 +
>  arch/x86/kernel/cpu/resctrl/core.c | 68 ++++++++++++++++++++++++++++++
>  2 files changed, 69 insertions(+)
> 
> diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
> index 1d111350197f..393d1b047617 100644
> --- a/arch/x86/include/asm/msr-index.h
> +++ b/arch/x86/include/asm/msr-index.h
> @@ -1100,6 +1100,7 @@
>  #define MSR_IA32_QM_CTR			0xc8e
>  #define MSR_IA32_PQR_ASSOC		0xc8f
>  #define MSR_IA32_L3_CBM_BASE		0xc90
> +#define MSR_RMID_SNC_CONFIG		0xca0
>  #define MSR_IA32_L2_CBM_BASE		0xd10
>  #define MSR_IA32_MBA_THRTL_BASE		0xd50
>  
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index ed4f55b3e5e4..9f0ac9721fab 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -16,11 +16,14 @@
>  
>  #define pr_fmt(fmt)	"resctrl: " fmt
>  
> +#include <linux/cpu.h>
>  #include <linux/slab.h>
>  #include <linux/err.h>
>  #include <linux/cacheinfo.h>
>  #include <linux/cpuhotplug.h>
> +#include <linux/mod_devicetable.h>
>  
> +#include <asm/cpu_device_id.h>
>  #include <asm/intel-family.h>
>  #include <asm/resctrl.h>
>  #include "internal.h"
> @@ -724,11 +727,34 @@ static void clear_closid_rmid(int cpu)
>  	wrmsr(MSR_IA32_PQR_ASSOC, 0, 0);
>  }
>  
> +/*
> + * The power-on reset value of MSR_RMID_SNC_CONFIG is 0x1
> + * which indicates that RMIDs are configured in legacy mode.
> + * Clearing bit 0 reconfigures the RMID counters for use
> + * in Sub NUMA Cluster mode.
> + */

I think I missed where "legacy mode" and "Sub NUMA Cluster mode"
are explained. 

> +static void snc_remap_rmids(int cpu)
> +{
> +	u64 val;
> +
> +	/* Only need to enable once per package */

Sentence end with period.

> +	if (cpumask_first(topology_core_cpumask(cpu)) != cpu)
> +		return;
> +

This is an area I am not familiar with. The above code seems
to assume that CPUs are onlined in a particular numerical
order. For example, if I understand correctly, if CPUs
are onlined from higher number to lower number then
the above code may end up running on every CPU online.

> +	rdmsrl(MSR_RMID_SNC_CONFIG, val);
> +	val &= ~BIT_ULL(0);
> +	wrmsrl(MSR_RMID_SNC_CONFIG, val);
> +}
> +
>  static int resctrl_online_cpu(unsigned int cpu)
>  {
>  	struct rdt_resource *r;
>  
>  	mutex_lock(&rdtgroup_mutex);
> +
> +	if (snc_nodes_per_l3_cache > 1)
> +		snc_remap_rmids(cpu);
> +
>  	for_each_capable_rdt_resource(r)
>  		domain_add_cpu(cpu, r);
>  	/* The cpu is set in default rdtgroup after online. */
> @@ -983,11 +1009,53 @@ static __init bool get_rdt_resources(void)
>  	return (rdt_mon_capable || rdt_alloc_capable);
>  }
>  
> +/* CPU models that support MSR_RMID_SNC_CONFIG */
> +static const struct x86_cpu_id snc_cpu_ids[] __initconst = {
> +	X86_MATCH_INTEL_FAM6_MODEL(ICELAKE_X, 0),
> +	X86_MATCH_INTEL_FAM6_MODEL(SAPPHIRERAPIDS_X, 0),
> +	X86_MATCH_INTEL_FAM6_MODEL(EMERALDRAPIDS_X, 0),
> +	{}
> +};
> +
> +static __init int get_snc_config(void)

Could this function please get a comment about what it does?
The first paragraph from the commit message should be ok.

> +{
> +	unsigned long *node_caches;
> +	int mem_only_nodes = 0;
> +	int cpu, node, ret;
> +
> +	if (!x86_match_cpu(snc_cpu_ids))
> +		return 1;
> +
> +	node_caches = kcalloc(BITS_TO_LONGS(nr_node_ids), sizeof(*node_caches), GFP_KERNEL);

How about bitmap_zalloc()?

> +	if (!node_caches)
> +		return 1;
> +
> +	cpus_read_lock();
> +	for_each_node(node) {
> +		cpu = cpumask_first(cpumask_of_node(node));
> +		if (cpu < nr_cpu_ids)
> +			set_bit(get_cpu_cacheinfo_id(cpu, 3), node_caches);
> +		else
> +			mem_only_nodes++;
> +	}
> +	cpus_read_unlock();
> +
> +	ret = (nr_node_ids - mem_only_nodes) / bitmap_weight(node_caches, nr_node_ids);

My static checker complains about the above line risking
a "divide by zero" that may be the case if the bitmap_weight() returns
zero. You may need to ensure this is safe here before more static checkers
start analyzing this code.

> +	kfree(node_caches);
> +
> +	if (ret > 1)
> +		rdt_resources_all[RDT_RESOURCE_L3].r_resctrl.mon_scope = RESCTRL_NODE;
> +
> +	return ret;
> +}

As discussed there are scenarios where the above may not provide the
correct value. Could you please document the assumptions of this code to
highlight this to x86 maintainers for their opinions?

> +
>  static __init void rdt_init_res_defs_intel(void)
>  {
>  	struct rdt_hw_resource *hw_res;
>  	struct rdt_resource *r;
>  
> +	snc_nodes_per_l3_cache = get_snc_config();
> +
>  	for_each_rdt_resource(r) {
>  		hw_res = resctrl_to_arch_res(r);
>  


Reinette