Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <a907fe31-3877-c4f0-7acf-785c28fd2e1f@intel.com>
Date:   Tue, 18 Jul 2023 13:42:20 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.13.0
Subject: Re: [PATCH v3 4/8] x86/resctrl: Add code to setup monitoring at L3 or
 NODE scope.
Content-Language: en-US
To:     Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>,
        "Peter Newman" <peternewman@google.com>,
        Jonathan Corbet <corbet@lwn.net>,
        Shuah Khan <skhan@linuxfoundation.org>, <x86@kernel.org>
CC:     Shaopeng Tan <tan.shaopeng@fujitsu.com>,
        James Morse <james.morse@arm.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Babu Moger <babu.moger@amd.com>,
        Randy Dunlap <rdunlap@infradead.org>,
        <linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
        <patches@lists.linux.dev>
References: <20230713163207.219710-1-tony.luck@intel.com>
 <20230713163207.219710-5-tony.luck@intel.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20230713163207.219710-5-tony.luck@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?U3ZYYmhiSzRVTVNTRFUvNzF6Qy9LQ2hGQVN3ZHplVnU1SXdQOEVDY0lLcEwv?=
 =?utf-8?B?T3hmTm4xV2tGc0I3cFpibERnanFraXFGZXhQWURMU0xnd1pwZktrNURmcGdo?=
 =?utf-8?B?cEdVMWRsaHJWaWdSV3I1VEtxcjZEZ0FKSklsbnh6WllXTldVVGt3a2FNMWhh?=
 =?utf-8?B?WHFFSzJBdzcrNXJzTVlKUzMyNnZhWE5EMC9OcWVvcjZHZGxFSXJxSG1GeXN4?=
 =?utf-8?B?S0dZZ1dpeXIvaEE3bytmMk5RNkFsR2liNEhHSWJMSE8rQ0xhRXY0NWUrS2VO?=
 =?utf-8?B?TzFIcUxKWHRRS3lhM2N3Q2M5dkRRUmdpaUwxU2xBVzdwTUsraGFEbGZ3dFU0?=
 =?utf-8?B?dXB6TkRCYzZ2UnlaamdNbTVzNXVKWDNuVW5mRERzYzNveHZPQ093K0ZVVVZV?=
 =?utf-8?B?REt1azlUd2p3Y1hkZldYaEdtWTVOSTg2Wk5QN1JhZXNuajdPRXovQjhheTdG?=
 =?utf-8?B?d3VLTnlyOHE0OGpIaFEzeDdJQ0FwUkpkWXlTOUs5ZjJVVUR0aXdKa2NXWVlI?=
 =?utf-8?B?d1d2MDlsNjdoQWswcU1BeUh6dHdwenZQOVJiR2pKOTZVRDBiTzBLV205ME1v?=
 =?utf-8?B?VldKaXVUSDJGdmpZOHJpK2RuaXlIcXpCSmh0TEhkdXh6WnBIWEhYdGgycHZn?=
 =?utf-8?B?eVRJZzVWbGpWVW14d2doTnFUZ2I3eEMxWW1nM2M2YjR2U2swV1M3eUtoQ2s4?=
 =?utf-8?B?di9uRTAyUmNnVjBRZDBGM2dLdnFzQ3ZTbFlzSitRVmo5OUNTcEs0d3pVem9j?=
 =?utf-8?B?eU9jZmdNUFdHNnUxRERpL2g1aGZrUTZDRlVHTmIyRHg2WkF1SlpsbFhDOERK?=
 =?utf-8?B?bGlVUnVPek4wWjVaUWFRVjRKSkhzUkt3b215VEx6YVdVMDBmRC9oS0tJLzVK?=
 =?utf-8?B?WlVkcGg4L3ZhZE5HZVJpTTlWaEErNFRaVEMvSVh6akJhVEIyN0Q3YVpmN1p2?=
 =?utf-8?B?Y3NUM1VUVkhueHJsNTI1SXNUWUVyUFkweHBvbk4rcHUwcGtrWjhwRVZwUzh3?=
 =?utf-8?B?NlRtRkxLN0lYL011Q1hmbFBDRkljR3g0aTNDeEdPU3hFSGJpL1BsVzJvWngv?=
 =?utf-8?B?ZFh4MFBNM3d1SXdHY3BPNmJTMjg4U3gwUFB0Uk5lY0ppS1BGNkN5UStvSEpO?=
 =?utf-8?B?KzJuNXEwZVVwaGl3dG1lbmFHbWJZSTRiTGd0T3hIRWVWUzgveEZ1MC8wcDQy?=
 =?utf-8?B?T3hKSkp1STVFc3Q2VE9VeVpvT1FTZ3UxY21nMDZSM0paYVBMaW1IVTR2WUc4?=
 =?utf-8?B?R21JejNzOHVSUCsveUNGTHVxSXNITXd0VnY2MG9RWWx1eWxMWmJxNzBpOWNP?=
 =?utf-8?B?VHI1UzBMNUwwSWtUNjdsUGRWMGNnV0JWcFFCcDVidmRvWjdZdFRRUW4wZTND?=
 =?utf-8?B?NE1DbFNaVk1QUy8rR2ZuS0grd21kUGUrWGlLUWU3QUZqZjQ5SDNCZmRlS3VL?=
 =?utf-8?B?RXFWNEVHY21hQ3RkTXFZY2pNcm5QQzVOem5zcGZybXM2N3dUTXQvRTMwLzgv?=
 =?utf-8?B?Snl6L0RqaXQ4SjJzNUNQNHgvRGo4UUN3MmpsTk5aYnU5WFVWQUcrUmIrMHJF?=
 =?utf-8?B?QkpaRlNUYU1IYnh2NFNMVU1BY0xLeFN4SGd1dmhHLzhQaDdZbDBNU1lZV21a?=
 =?utf-8?B?eUVHOFRiN0FOZjlQSGdIRmIrQVd0eHdKNnBOemgrcytwSkpIOTloMytUbHFI?=
 =?utf-8?B?R2ZENmk3NWtPcnlpbE04NzlqRVFCb0R1b2VUSGRWUXpJTzBuK0dKclV4dEg0?=
 =?utf-8?B?WTRjMzA3OElhUjJPZExTaDBncW16RjNua0dwcGlnUUliaCtKSGNLdDhCL3ZK?=
 =?utf-8?B?aUY1UnR6SVg4bHE3OUEra3pQUXdQUGhGR003aEw1SEZNdjM4M0lFRkdWa3Ju?=
 =?utf-8?B?UjZTWmJYa3RhYW83VnVLbm9FSnhyWVFsU1NQZ3QvNm1DSmhINSszb2ViUXlV?=
 =?utf-8?B?aFMyWi81UzZPQVdHN1JBVE1pazdDeXhHVmlVeFg2QXJCS0UrS051alFOcGVX?=
 =?utf-8?B?TVM4cUMvYS9EMUloRE4rQ3RGQUNDL3VqdHlkRUVKT3FXazRKbUlEWTNuK1Fo?=
 =?utf-8?B?eUQ4ak5ZYml5MThnZmRxK0V1SjM5RmJoY0RsUWRWaVVueFl4cjlmck05VWJl?=
 =?utf-8?B?cDRuMGN3bERqS3FETUxDdGdJenl1MUl2T1BsNUxPQ0RpTlY1RE1RbXh5bHls?=
 =?utf-8?B?Zmc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 622fd83c-9ec5-461c-376e-08db87cf7d26
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2023 20:42:23.2142
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 926ACVRdl1k4WZTrsaMpT8J7/nRh3jUyMFcQRLpnwQj0kQkTBNJFnNthldNCGQo/SG0JMx6b0/DRevj2AMf8OnXkrcEmLdvnYaymdDGW+LY=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7890
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Tony,

Regarding subject: "Add code" is not necessary.

On 7/13/2023 9:32 AM, Tony Luck wrote:
> When Sub-NUMA cluster is enabled (snc_ways > 1) use the RDT_RESOURCE_NODE
> instead of RDT_RESOURCE_L3 for all monitoring operations.

This duplication of resource does not look right to me.
RDT_RESOURCE_NODE now contains the monitoring data for RDT_RESOURCE_L3
with related structures within RDT_RESOURCE_L3 going unused.

> 
> The mon_scale and num_rmid values from CPUID(0xf,0x1),(EBX,ECX) must be
> scaled down by the number of Sub-NUMA Clusters.
> 
> A subsequent change will detect sub-NUMA cluster mode and set
> "snc_ways". For now set to one (meaning each L3 cache spans one
> node).
> 
> Signed-off-by: Tony Luck <tony.luck@intel.com>
> Reviewed-by: Peter Newman <peternewman@google.com>
> ---
>  arch/x86/kernel/cpu/resctrl/internal.h | 7 +++++++
>  arch/x86/kernel/cpu/resctrl/core.c     | 7 ++++++-
>  arch/x86/kernel/cpu/resctrl/monitor.c  | 4 ++--
>  arch/x86/kernel/cpu/resctrl/rdtgroup.c | 2 +-
>  4 files changed, 16 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
> index 243017096ddf..38bac0062c82 100644
> --- a/arch/x86/kernel/cpu/resctrl/internal.h
> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
> @@ -430,6 +430,8 @@ DECLARE_STATIC_KEY_FALSE(rdt_alloc_enable_key);
>  
>  extern struct dentry *debugfs_resctrl;
>  
> +extern int snc_ways;
> +
>  enum resctrl_res_level {
>  	RDT_RESOURCE_L3,
>  	RDT_RESOURCE_L2,
> @@ -447,6 +449,11 @@ enum resctrl_scope {
>  	SCOPE_NODE,
>  };
>  
> +static inline int get_mbm_res_level(void)
> +{
> +	return snc_ways > 1 ? RDT_RESOURCE_NODE : RDT_RESOURCE_L3;
> +}

Need to return the enum here? It may be simpler for this helper to
just return a pointer to the resource. (Although the need for a
separate resource is still not clear to me.)

> +
>  static inline struct rdt_resource *resctrl_inc(struct rdt_resource *res)
>  {
>  	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(res);
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index e4bd3072927c..6fe9f87d4403 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -48,6 +48,11 @@ int max_name_width, max_data_width;
>   */
>  bool rdt_alloc_capable;
>  
> +/*
> + * How many Sub-Numa Cluster nodes share a single L3 cache
> + */
> +int snc_ways = 1;
> +

Since snc_ways is always used I think the comment should provide
more detail on the possible values it may have. For example, to
a reader it may not be obvious what the value of snc_ways should be
if SNC is disabled. Also, what does "ways" refer to? (Also mentioned
later).

>  static void
>  mba_wrmsr_intel(struct rdt_domain *d, struct msr_param *m,
>  		struct rdt_resource *r);
> @@ -831,7 +836,7 @@ static __init bool get_rdt_alloc_resources(void)
>  
>  static __init bool get_rdt_mon_resources(void)
>  {
> -	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
> +	struct rdt_resource *r = &rdt_resources_all[get_mbm_res_level()].r_resctrl;
>  
>  	if (rdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC))
>  		rdt_mon_features |= (1 << QOS_L3_OCCUP_EVENT_ID);
> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
> index 9be6ffdd01ae..da3f36212898 100644
> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
> @@ -787,8 +787,8 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
>  	int ret;
>  
>  	resctrl_rmid_realloc_limit = boot_cpu_data.x86_cache_size * 1024;
> -	hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale;
> -	r->num_rmid = boot_cpu_data.x86_cache_max_rmid + 1;
> +	hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale / snc_ways;
> +	r->num_rmid = (boot_cpu_data.x86_cache_max_rmid + 1) / snc_ways;
>  	hw_res->mbm_width = MBM_CNTR_WIDTH_BASE;
>  
>  	if (mbm_offset > 0 && mbm_offset <= MBM_CNTR_WIDTH_OFFSET_MAX)
> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> index 418658f0a9ad..d037f3da9e55 100644
> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> @@ -2524,7 +2524,7 @@ static int rdt_get_tree(struct fs_context *fc)
>  		static_branch_enable_cpuslocked(&rdt_enable_key);
>  
>  	if (is_mbm_enabled()) {
> -		r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
> +		r = &rdt_resources_all[get_mbm_res_level()].r_resctrl;
>  		list_for_each_entry(dom, &r->domains, list)
>  			mbm_setup_overflow_handler(dom, MBM_OVERFLOW_INTERVAL);
>  	}

This final hunk makes me wonder why the monitor.c:
mon_resource is necessary at all. A single helper used everywhere
may be simpler.

Reinette