Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32;
Message-ID: <7857bcde-effc-43ea-8df9-db79f6d396a3@intel.com>
Date:   Mon, 6 Nov 2023 16:33:08 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v10 7/8] x86/resctrl: Sub NUMA Cluster detection and
 enable
Content-Language: en-US
To:     Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>,
        "Peter Newman" <peternewman@google.com>,
        Jonathan Corbet <corbet@lwn.net>,
        "Shuah Khan" <skhan@linuxfoundation.org>, <x86@kernel.org>
CC:     Shaopeng Tan <tan.shaopeng@fujitsu.com>,
        James Morse <james.morse@arm.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Babu Moger <babu.moger@amd.com>,
        Randy Dunlap <rdunlap@infradead.org>,
        <linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
        <patches@lists.linux.dev>
References: <20231020213100.123598-1-tony.luck@intel.com>
 <20231031211708.37390-1-tony.luck@intel.com>
 <20231031211708.37390-8-tony.luck@intel.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20231031211708.37390-8-tony.luck@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SUxmWEdIOGYrUmtjdXpudXEwRWN3NnRIZlIxZGJaNnZzWlZtZEZ5YyttYkRQ?=
 =?utf-8?B?VGIvTGdpN2FZRWtwbitvMmk5YXV2VWo3TVAzVk5qNFlaNWIySmhQdXI2Rlls?=
 =?utf-8?B?eGZtWnR5YTZFVEF0ejdmcHY3YXhqSTdUWFlOYzEvM25UK3hxUEdQaHVHWXNz?=
 =?utf-8?B?cFV0SXlZNGdHSWNNSERWTzBXWWdScmJXT3g5NjFHN0duKzZZZ0dWSmZjTnAy?=
 =?utf-8?B?dTVzVUpFNUxjQWpoMlliOC9sbFlvdFB6Tnhka0k2QVlDWHMyODdQZHZXdGZP?=
 =?utf-8?B?YVoxNGhoam5ubjcxYklmNjExWm1xZUhMczZLZmcySExWbWFuZmdnNnh6VzNk?=
 =?utf-8?B?UjhtRXo3ZEVYZ1grRm9WSGZubEJHSVJQUHdHYmtjNHFrM0Y0VGNLRG11RFJt?=
 =?utf-8?B?ZWppV0cySjdjZkRLSXFQZm1keTJSRFRzT2FWemluaGI0SG5nVGdmNDk3akd2?=
 =?utf-8?B?OFBVRkEzQXJOWkhGbFBZZ1NXeGs0d1NlbVNjdlZoek1lbC9WcHV4RzNVa2xp?=
 =?utf-8?B?Q0RmM2RSK3FBNDhkNUx5R1hOY0NIaWFYc2xhTWpwaWpRTTUyTk44ZGJDUGhq?=
 =?utf-8?B?bDBUdDZkcEZ0ckQ5WnRKTklRYkxwMHNIQjkveDczUTJ6YnVmZDRqRkwyaEow?=
 =?utf-8?B?cWtwK2NUWmcxcTBSbUpTRlBlRGxCTlc5V2FDLzNpemVWZkluYkhDeUNIZldC?=
 =?utf-8?B?UzE4MDlkNXhqVzg2RjE4OVJhZGlqV0cxZUdKVUlWR0NQWFhOR3FWZi9Nem53?=
 =?utf-8?B?U1dqMEZxamswTGVOZzlwTi9uSGU0Y3NpVHZyWDJyY240WUtlMUdzSExCSW5h?=
 =?utf-8?B?UWZWZ3FIK3o3UXZxeFVvVnBtWU1YUksvNGFKNElDNVdPQ1Izazk2d09BMGJu?=
 =?utf-8?B?QUU1cGRUSFkxZ1pQRlYzY2dVYTgzMnVxZUVlaHBjNW9ZOWtBU3BXMXlKMVJ1?=
 =?utf-8?B?anNXUFA3UFVLUHdidjkrR0JQVkpYMVpJWjlYUkFzU0NXcHY2VEcwYmhqRzBh?=
 =?utf-8?B?bmtyMXZuTFFoYitRUlRldW5ZbUlHMWxYRW5CaE9vdk42T0tySmxmMG4xcWg4?=
 =?utf-8?B?bXF5cmdvaEpnallJTllVU0hmaGJ6aVhQNm1uTlB2MXVjUDBibTNFR0dGOUFP?=
 =?utf-8?B?R1RhNmFiMmdvQlc4OGswNktKWWR1eGtGNXZQL2RadjVMM29Qb3RudzhRaG82?=
 =?utf-8?B?U1MwQmFKempoZmt2RVlZTnpoeEZ6KzNia0NKVGp4ZngrV2lYam5Ga3FQc2dH?=
 =?utf-8?B?c09tOVZZYVhqUkxSYUdPbXhFVCsreUJOdXJ1UHJpdTBsaTNmbTM0NWkrdmpP?=
 =?utf-8?B?dmNqT041SjRsd3hLTkZMQXkwTm5zQ3Q4TVZaQWErUExWY3AwR2NrT2ZnTDFn?=
 =?utf-8?B?eUwrM21MdkZacGhSYnZjazY4WkJRdkthc3p3bkxtOW5ZUUdzR0FEeC9XYzU1?=
 =?utf-8?B?ZjV6d2tOUjZIZi9ub2FCajE1U25XYlEzTGVWZ09rNWlJeXJya2psSFhwcWRE?=
 =?utf-8?B?em5NeGdpQU1CUjMvUG1GMVNqRm1GejE0bWpuOS9TYVlpaWFFTnhMTmhOTjZV?=
 =?utf-8?B?dVV0Wjh0aFk4Vk9Ic3R5V2JyUXJPdG42akVsSlNJNnZqdG4vSk1NTmc0OGdV?=
 =?utf-8?B?TXhNaHgrZXJweE1mNE5ZY3BVdDllaHNUYkVzMUFXUHA1QUlsL0hNbjZ0OWF2?=
 =?utf-8?B?Mm5CcVRMQjJOVEhPcmRoUERHdG9UYzZhYUwvaXZ3MU1hUm9zV3dCdUhRelBt?=
 =?utf-8?B?Qm04ZmhjbXE4SU5ZYlk2TlNTRW42WUVMSmZndVlrdnBPL0ZLWWc1MllyVHVz?=
 =?utf-8?B?dmFaZVlTdWJKWmdrNVdBS0FlZmRiNy91TkpVa1djdWQzRzhVNVgrSmtCR0Fk?=
 =?utf-8?B?dS9OZWNOZ0Q5OXB1SS9uUzYrR2owRFoxWWtQb2NFQm1FdTJIb0NwaWhqWmln?=
 =?utf-8?B?bHR0SmxKNE5YS1QwbEQxY0JUSVpiYVBKYVludDJiV0NaS3EwWEhBKzFlVTNs?=
 =?utf-8?B?YW0zVmFJSy9KWjBQTzBWTUxtWXN0bS82RUJkYmROdmY4Q0YrZng1K1NFQzUy?=
 =?utf-8?B?cEtKZ2xFSnlaaDVqdGhBWXZERUVBU1YxcmJadnhKeXVzbzNMSVgzd1lIbCt5?=
 =?utf-8?B?dHZuTDZDTTFxMHFNbGc0SkZld2pWanV1VHVHcFYzMGhTWFhKaWYwSG1Id05R?=
 =?utf-8?B?VFE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 464adf32-be0e-4203-2881-08dbdf291dd4
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2023 00:33:09.1444
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1J9uovwluOHyZMLKnw0I0eZ3O7WwJe1uGfjTCr5VYYcptdwRHFvVLShBHgTewUtce68Yz+7EOjqx7PaiduGqEUMbJZRpT07vqtv+nO+MM2o=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB7614
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 06 Nov 2023 16:33:30 -0800 (PST)

Hi Tony,

On 10/31/2023 2:17 PM, Tony Luck wrote:
> There isn't a simple hardware bit that indicates whether a CPU is
> running in Sub NUMA Cluster (SNC) mode. Infer the state by comparing
> the ratio of NUMA nodes to L3 cache instances.
> 
> When SNC mode is detected, reconfigure the RMID counters by updating
> the MSR_RMID_SNC_CONFIG MSR on each socket as CPUs are seen.
> 
> Clearing bit zero of the MSR divides the RMIDs and renumbers the ones
> on the second SNC node to start from zero.
> 
> Signed-off-by: Tony Luck <tony.luck@intel.com>
> ---
> Changes since v9
> Expand h/w to hardware (commit and code comments)
> Remove "earlier commit" reference
> s/counnter/counter/
> Check for offline CPUs and warn user SNC detection may be broken.
> 
>  arch/x86/include/asm/msr-index.h   |   1 +
>  arch/x86/kernel/cpu/resctrl/core.c | 100 ++++++++++++++++++++++++++++-
>  2 files changed, 99 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
> index e3fa9cecd599..4285a5ee81fe 100644
> --- a/arch/x86/include/asm/msr-index.h
> +++ b/arch/x86/include/asm/msr-index.h
> @@ -1109,6 +1109,7 @@
>  #define MSR_IA32_QM_CTR			0xc8e
>  #define MSR_IA32_PQR_ASSOC		0xc8f
>  #define MSR_IA32_L3_CBM_BASE		0xc90
> +#define MSR_RMID_SNC_CONFIG		0xca0
>  #define MSR_IA32_L2_CBM_BASE		0xd10
>  #define MSR_IA32_MBA_THRTL_BASE		0xd50
>  
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 97d2a5a7dd41..034f9797e1fb 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -16,11 +16,14 @@
>  
>  #define pr_fmt(fmt)	"resctrl: " fmt
>  
> +#include <linux/cpu.h>
>  #include <linux/slab.h>
>  #include <linux/err.h>
>  #include <linux/cacheinfo.h>
>  #include <linux/cpuhotplug.h>
> +#include <linux/mod_devicetable.h>
>  
> +#include <asm/cpu_device_id.h>
>  #include <asm/intel-family.h>
>  #include <asm/resctrl.h>
>  #include "internal.h"
> @@ -184,10 +187,10 @@ bool is_mba_sc(struct rdt_resource *r)
>  
>  /*
>   * rdt_get_mb_table() - get a mapping of bandwidth(b/w) percentage values
> - * exposed to user interface and the h/w understandable delay values.
> + * exposed to user interface and the hardware understandable delay values.
>   *
>   * The non-linear delay values have the granularity of power of two
> - * and also the h/w does not guarantee a curve for configured delay
> + * and also the hardware does not guarantee a curve for configured delay
>   * values vs. actual b/w enforced.
>   * Hence we need a mapping that is pre calibrated so the user can
>   * express the memory b/w as a percentage value.

This seems out of place here. If you want to make such a global change
it can be done as a separate patch. For this work it can just consistently
use "hardware" in own areas changed.

Reinette