Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34;
Message-ID: <1c1a16a5-f235-4179-9d0f-1556e11d9c11@intel.com>
Date:   Thu, 30 Nov 2023 13:47:10 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v12 7/8] x86/resctrl: Sub NUMA Cluster detection and
 enable
Content-Language: en-US
To:     Tony Luck <tony.luck@intel.com>, Fam Zheng <fam@euphon.net>
CC:     Fenghua Yu <fenghua.yu@intel.com>,
        Peter Newman <peternewman@google.com>,
        Jonathan Corbet <corbet@lwn.net>,
        Shuah Khan <skhan@linuxfoundation.org>, <x86@kernel.org>,
        Shaopeng Tan <tan.shaopeng@fujitsu.com>,
        James Morse <james.morse@arm.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Babu Moger <babu.moger@amd.com>,
        Randy Dunlap <rdunlap@infradead.org>,
        <linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
        <patches@lists.linux.dev>,
        Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
References: <20231109230915.73600-1-tony.luck@intel.com>
 <20231130003418.89964-1-tony.luck@intel.com>
 <20231130003418.89964-8-tony.luck@intel.com> <ZWjOBw0Ygyw226Cc@dell>
 <ZWj3NdI/qLNOgyg0@agluck-desk3>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <ZWj3NdI/qLNOgyg0@agluck-desk3>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?V0wyaVovRzM1KzZjdm9yWHBzcXZhUTIwckwwWmxPT2ZWUzJ5T2FPM3lRelRM?=
 =?utf-8?B?VnRlMFFoLzFyKzA1bGFuQWFZbkozYnBYQ2Z3bFdQRENVakR3SC80a01nS2Rp?=
 =?utf-8?B?cFF2YkhoWHNsUTNPNkZWWjdvVW5Ld0kyWGFmdlkwT3M3KytvenIxeHM4ZXRT?=
 =?utf-8?B?RUVzT2R5TTl2QTAvZGFBdUtCOWo0QndDQ21sc1QyRGtISTNlWXQ5cW81N1Vu?=
 =?utf-8?B?bEVENjhWdnZCYi9YWnhFbVM0b09QUHBlb1huMlpXa2swTVltVjI0TGM2Ym5z?=
 =?utf-8?B?aEVGUkRVWXREOFhFdjZrRGJSOHR6TjFzd2xWSzB2ZjdkL1BGNHlXZm5uKzd3?=
 =?utf-8?B?RDkyMTRNd09IQzZ6cXR6U3E4NGd6RmlSWW9xNDY4aTFSU2VWU1FFVEEzOVF1?=
 =?utf-8?B?RTJZcnJDS3pRV251aXpsWWRITlN5MWxaNWxqMzlNOEtJOVhRZVl3bUNCK3ds?=
 =?utf-8?B?MnY3SktiNlhMeXRzdGtrUzRrakxibEVwejZCTkI2S29NYnF3M29CUXVoOVIr?=
 =?utf-8?B?QWNQZllIWTM3cm82aHBIUUhwNVZFMktuVDJpSDZSYUlhN3ROeE54ZGdSbGVu?=
 =?utf-8?B?ayt1UlBveW9jZ1p2WDRlQUJVY1Q4TVNBb21pZ05WcW16OWhYOW8wWFZZc2dK?=
 =?utf-8?B?NWZnRDN1dXk4WVNYUFJjRE1qdUJKKzBMRlIzV3JkR082Tkx5V0dZT2paSkd1?=
 =?utf-8?B?N1ZLV1VJTE0vU3NvNjYwNmFQZEs0VE8yZjNTeEJRZFVkZjZ0b0ZoWlpjZHJ6?=
 =?utf-8?B?eEFQdU1JYUkwSU9yVkdxRk9GSXUyYlAyVGpKTzE3aVFFTlljTmJYeTh2SkNr?=
 =?utf-8?B?RjdNK1NpSWgrMXBZeU5zV2pqWjhZME1LUW9iZzJzSEcvRDNmaU5DTTNaRXBI?=
 =?utf-8?B?MVd6aTNxd1h1bDB2YWIxUmlydzBUQXFjSmZZV0oxUVprMk1SaHVxcCsxVHUv?=
 =?utf-8?B?UGt1N0RRa25YelVVMXZhaWhjTkVVazVhM2c0bUlVVjJNN09QQlJnNGxYU2dk?=
 =?utf-8?B?VGw2TlVhSlI5aGlvVEVCOHRxNEVMdzNuVUowL213SzNwYW5UVUxBQVJZN3ZU?=
 =?utf-8?B?UUR5MUh1RjFtQVAzeVRFeWlNU2NWbk4xcSt3cUdQMUsyekVseUVObUNCS2lr?=
 =?utf-8?B?Y1lLT2JwV0Qyc2tsQmxEaTIxQzl3M0RUM05WVlZaR2dYWVF5aWtNcWJ1a2dk?=
 =?utf-8?B?TGVWeFJzSEdwU2VoWS9wak5pelV5cjd1ZGFTZFU1Tlk3bFptZnNtYVdWMTFW?=
 =?utf-8?B?eXVGclEvRmdYNGQxZWhJdVRRbzFKS1RMRlRuUnR5SVVSMkN4L2JmU1FmTExW?=
 =?utf-8?B?dEhCQ25PVVNZOFRJR3ZJV2FqZjJqK2NXbzM4ZGphZ1pVenJqRGpYR0xzUENM?=
 =?utf-8?B?bXEySnhCV1N3Sm0zL0FMZ0prMkFzY2srOFd4Z2p3M2NDU2lNUkIreUhFR2Z4?=
 =?utf-8?B?KzhnVHQ1VW5CTkdYc2JDVkFlOEdINjhldWZBQktDUFZTZVA2YWl6MGJ4RkFU?=
 =?utf-8?B?a3l5WWVCTDh4eS8xMkttUVRzeWlROS9tcDNqVDRkVytGLzVrUm5naXhYNFk0?=
 =?utf-8?B?UXJ2YWFidjlmSkZDUFRQSnpaNG1NMDlFbTRUTE5LNTBpbXZKMEkrZStJR1Fo?=
 =?utf-8?B?L1crVnF6d3Jtamcyb2t3TVFEY3dVVEl2aUVkZlZFSmlDaHJBR3FyaTVKTUlM?=
 =?utf-8?B?UjlxSnA1SkxBLzhOVGR5YThKOHdrV1FycHpWK0hTOHhwRE9tZDcvVWRCWm1B?=
 =?utf-8?B?MVhJQzlpUGNBTWU1RTA1WWVoWlVzcXBsVHR4Zlc1TnVWdTR4dkhLbDNvdkcw?=
 =?utf-8?B?ZUg5NkJCOGF3OUhXOXIweHlCUU5BVFpXUElPcFFkeXZwRTRXNzkyUmhKZHJ0?=
 =?utf-8?B?VzJoTTQ5dEJ4RUxvNVNxejRNTHJhV2RHbXZBZkhMc1REYnJ6dDd2N2p3SjdM?=
 =?utf-8?B?ZFZXMWJ3cW9DWEFyays2MjQ5TkNOQTNURHZoYUdWSFVlMFRBai9za0RueVRR?=
 =?utf-8?B?WGN5V3BjbHJyazNRK0V0ODAycExqdENESlV3MXhQNWMyNXF5c09Bb1ZnMGZa?=
 =?utf-8?B?SEVZOU40MXZsM05pd3JvRFlrdEpDcmQ5UzZ6aDNzekhGbGpIYlZXaVZKQklJ?=
 =?utf-8?B?Vnp3bHN3MC9NT3NmZ3BtWjM1aTFWeTl3WlBXUmQ0eWFYY1JyUVBaQVk2Rko0?=
 =?utf-8?B?ZUE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 285c1d5c-170b-4e89-c607-08dbf1ede9aa
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Nov 2023 21:47:13.4712
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: NGz+m40MG4Tfpe2GbsBSf8dPdd9N8p6abzrRBBwxKlRJ3IlYPdhEup9mp6TzNrZTqBau9avutc+fWTxTLt5EtbZeUoSMLjMp6nLwsTN2QcU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR11MB6089
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Thu, 30 Nov 2023 13:47:28 -0800 (PST)

Hi Tony,

On 11/30/2023 12:57 PM, Tony Luck wrote:
> On Thu, Nov 30, 2023 at 06:02:42PM +0000, Fam Zheng wrote:
>>> +static __init int snc_get_config(void)
>>> +{
>>> +	unsigned long *node_caches;
>>> +	int mem_only_nodes = 0;
>>> +	int cpu, node, ret;
>>> +	int num_l3_caches;
>>> +
>>> +	if (!x86_match_cpu(snc_cpu_ids))
>>> +		return 1;
>>> +
>>> +	node_caches = bitmap_zalloc(nr_node_ids, GFP_KERNEL);
>>> +	if (!node_caches)
>>> +		return 1;
>>> +
>>> +	cpus_read_lock();
>>> +
>>> +	if (num_online_cpus() != num_present_cpus())
>>> +		pr_warn("Some CPUs offline, SNC detection may be incorrect\n");
>>> +
>>> +	for_each_node(node) {
>>> +		cpu = cpumask_first(cpumask_of_node(node));
>>> +		if (cpu < nr_cpu_ids)
>>> +			set_bit(get_cpu_cacheinfo_id(cpu, 3), node_caches);
>>
>> Are we sure get_cpu_cacheinfo_id() is an valid index here? Looking at
>> the function it could be -1 or larger than nr_node_ids.
> 
> Fam,
> 
> Return -1 is possible (in the case where first CPU on a node doesn't
> have an L3 cache). Larger than nr_node_ids seems a bit more speculative.
> It would mean a system with multiple L3 cache instances per node. I
> suppose that's theoretically possible. In the limit case every CPU may
> have its own personal L3 cache, but still have multiple CPUs grouped
> together on a node.
> 
> Patch below (to be folded into part7 of next version). Increases the
> size of the bitmap. Checks for get_cpu_cacheinfo_id() returning -1.
> Patch just ignores the node in this case. I'm never quite sure how much
> code to add for "Can't happen" scenarios.
> 

Thank you.

> -Tony
> 
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 3293ab4c58b0..85f8a1b3feaf 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -1056,12 +1056,13 @@ static __init int snc_get_config(void)
>  	unsigned long *node_caches;
>  	int mem_only_nodes = 0;
>  	int cpu, node, ret;
> +	int cache_id;
>  	int num_l3_caches;

Please do maintain reverse fir order.

>  
>  	if (!x86_match_cpu(snc_cpu_ids))
>  		return 1;

I understand and welcome this change as motivated by robustness. Apart
from that, with this being a model specific feature for this particular
group of systems, it it not clear to me in which scenarios this could
run on a system where a present CPU does not have access to L3 cache.

>  
> -	node_caches = bitmap_zalloc(nr_node_ids, GFP_KERNEL);
> +	node_caches = bitmap_zalloc(num_online_cpus(), GFP_KERNEL);

Please do take care to take new bitmap size into account in all
places. From what I can tell there is a later bitmap_weight() call that
still uses nr_node_ids as size.

>  	if (!node_caches)
>  		return 1;
>  
> @@ -1072,10 +1073,13 @@ static __init int snc_get_config(void)
>  
>  	for_each_node(node) {
>  		cpu = cpumask_first(cpumask_of_node(node));
> -		if (cpu < nr_cpu_ids)
> -			set_bit(get_cpu_cacheinfo_id(cpu, 3), node_caches);
> -		else
> +		if (cpu < nr_cpu_ids) {
> +			cache_id = get_cpu_cacheinfo_id(cpu, 3);
> +			if (cache_id != -1)
> +				set_bit(cache_id, node_caches);
> +		} else {
>  			mem_only_nodes++;
> +		}
>  	}
>  	cpus_read_unlock();
>  

Could this code be made even more robust by checking the computed
snc_nodes_per_l3_cache against the limited actually possible values?
Forcing it to 1 if something went wrong?

Reinette