Received-SPF: pass (google.com: domain of linux-kernel+bounces-186732-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <ccd5df99-4d7e-4224-a07e-3897e370b53e@intel.com>
Date: Wed, 22 May 2024 14:24:12 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v18 14/17] x86/resctrl: Sum monitor data acrss Sub-NUMA
 (SNC) nodes when needed
To: Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>, Maciej
 Wieczor-Retman <maciej.wieczor-retman@intel.com>, Peter Newman
	<peternewman@google.com>, James Morse <james.morse@arm.com>, Babu Moger
	<babu.moger@amd.com>, Drew Fustini <dfustini@baylibre.com>, Dave Martin
	<Dave.Martin@arm.com>
CC: <x86@kernel.org>, <linux-kernel@vger.kernel.org>,
	<patches@lists.linux.dev>
References: <20240515222326.74166-1-tony.luck@intel.com>
 <20240515222326.74166-15-tony.luck@intel.com>
Content-Language: en-US
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20240515222326.74166-15-tony.luck@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?blZVbjdIMEVqcnNCUWNpUmthcTFRQ3gvNHo4OXNZMTNIamQvekVlT2RvdHBD?=
 =?utf-8?B?MkIxVEVFT1FJTVZhMjdnWURiRUVpQnlQc1BKRERUQi9ubWlZbVRXYzVOYlhQ?=
 =?utf-8?B?elFzYVZPMjlwbVdzdDBBMGxtdFlXeWMvbG9HQi93dk41b09wLzlpMVZKcjBO?=
 =?utf-8?B?MW1kNGVVWEZLZlBkeklHaTdUT25rSCthM0NiNCtVck9CVXd3WGgxazZEWnRZ?=
 =?utf-8?B?YnZPTDJRMkl6eGVYeEVJMU56ckN4Z2ZFSlhIb1FWR3c1c25XYmtVVzJhS0hZ?=
 =?utf-8?B?aU42TXhkK3Rpa2pXQTBJelRwRUkzMHQ0L3NCZnFpTVlDZ2VNSFlkQ3UrSmtC?=
 =?utf-8?B?RU1xZzVlb2ZURklHMVBvRjZMOHdnd3JyVkJWOVN4MDZ1aEpiSTNLQURDM1BB?=
 =?utf-8?B?NHZoRUUwbGxuVEFncXI2bkszUVI2dFkrVUpuNHlWcThEWVVHMUNrMkpJUDNx?=
 =?utf-8?B?bG54OWFGQWs0SXZQZmd5ZVVaNUJmVFk2RHpEdERVK1NMRDdWTXlWdGU0ODZS?=
 =?utf-8?B?emRjUmZsVkVTLzhadGhuM1IzUUc4Z2ExRjJQOHg1bXViSjY2bGtYdXQvcmE1?=
 =?utf-8?B?eGpvU21VVTJ0QjRaVklzUmMyOGRlTHc5OExGYkk0VnJud0F2RG5xZFFDQjBN?=
 =?utf-8?B?Q3pOeVJEa3RGRXo3YXdTRWdHNE1waEROZUdpZUV5RElTaG5BbUNjQms5ZmFO?=
 =?utf-8?B?Ty9laUR3SlJ6QThFZCtoTXlFS2Z0aWxlUk96N2o5RGp1b0NIek1VRS83Ulhu?=
 =?utf-8?B?clB5bW4vNlhldXp1TG1zanlJVlJ1b2pjWkRoMFVXaTBKbE5VY0JtTWNzUytX?=
 =?utf-8?B?bHRmZGdEbFlLOEZycGJnWTEzeWtCOGN2V0V1ZWtpV2FXVXRqRWVWVHNBWWF6?=
 =?utf-8?B?c3RsWkw0clVGL3A3OXpvSWt4M1JXbHRUaGlPQnpmRGgzcHRFL3JRS2JVQWhN?=
 =?utf-8?B?RXlVU1B6NjQva2ZPbnpYejJ6RHVUd0R5RWdjT1NLWmpCVXYvd2xEbDIwV1E0?=
 =?utf-8?B?WXJsZFhWR3d0NlBHTzFXcG5CTHRyUDFPNGFVbnI0QWhaUTh1U0YzT0pxcitT?=
 =?utf-8?B?WkJFTDlleHNocjlGVHBjbU5JNUEvdU14WGozWEhIY1RXZUlqNnRQWWZNSW1n?=
 =?utf-8?B?bWtRMXpzZllsa1JBUVlvdi8yTkd0alJxU0RML0xYZTRKMWFVbnA5VkFVbEpR?=
 =?utf-8?B?Mk1FaFdxTGhEdGJQckpTU2Q3NWF0TmROMUIzcXFWSmJwc3R2QjV5OEF2VlBJ?=
 =?utf-8?B?ajhLTmVtb3NWZDdNVWtabnRWQjRiZ2R3dVNxMjl5L3JDdmtENG4yaU1YMzV5?=
 =?utf-8?B?dU1SVzA1SGpPRmJtcStrMjN1Y3ltWTlSQ2xDcFJjV2ZYZmRRZ0YvQ2RTOEkz?=
 =?utf-8?B?U0VJZmgvZmRPc2JIbzVDbUQ4OTJBQW05bWlILzVJSVpNajJ3aksySVhOcmZU?=
 =?utf-8?B?SVNwVWtuK20wVDErUXRBRDlRR1hWN0xrd3lacjg4Z2lzdTRUbFlva3ZFckQz?=
 =?utf-8?B?MlQ1STNsMmlWZUVvZEpvaE9yalNlWXI5cThWa1loRTY2VjZDQUVBUFN2cEJn?=
 =?utf-8?B?cTVrQzJhL2FaaFMrb0pJbUtPMmcwUnYxbHZCbE1tdEU1SkM0ZTZFQXNBS1FK?=
 =?utf-8?B?WWZKUjRSb012MitiOG9WY0FZNG8xcmh5YkFia3IyOWlKNFVxYzZLYktxT0dK?=
 =?utf-8?B?cU9mak9oc3doVzFSMTFSeFJ0Lyt2RTJoMlZzVGtsckJFYjZ3QkRTMFVhc0Nu?=
 =?utf-8?B?K0FUcldPVjhCdDgzOFBEUXpGVm44T1B6V3VaK243a1Y2VE1KL2VVejg1YXVV?=
 =?utf-8?B?c1p4ZHErQnpXQ1lxQVFmaERIU0pqcVZ2ZTV6WkJpYVl3TEVjbFV1OUdBUUU1?=
 =?utf-8?B?WnNxY1haODZvMUNhN2doZjRKZDM1bXVTaGo4TU0xUEMzVGJBQ0d1K3l6UHRk?=
 =?utf-8?B?Q0ZFQnZOOGgraDZzS3lPRy9QOUdlQlVDbnF0bVRNTE91ejFORjdCWU94U3Ew?=
 =?utf-8?B?RWVhYnc4K21yV2tVZDdQclpIQmpwYWFhd05BUFR3MDlUOXhCTlZxZ0FEcEg0?=
 =?utf-8?B?dTdndHBhSk90cHBFQ3hVR2NIM0lubE1QajNBeUtrL1ZCOWtXOXBHN0U5MTNm?=
 =?utf-8?B?Q080UDFRdWZRVzFEWTZGMCttZ3kzb0hWdWxnTG5vQUxvRE1yNlA3Z1VmRCtH?=
 =?utf-8?B?Vmc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 62c8e8fb-7492-4510-3365-08dc7aa587b8
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 May 2024 21:24:14.7366
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: CTrlnYBinaEn7vFOEuM6DsbApkreE7wWMNx0vBiYBSt9OE08oS5NdnMZuHCiq7DWyBfA169uF/4UPC0Djz3AMqiY6JAfEAIyocvfK1EVZyQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB7056
X-OriginatorOrg: intel.com

Hi Tony,

shortlog: acrss -> across?

On 5/15/2024 3:23 PM, Tony Luck wrote:
> When the sumdomains fields is set in the rmid_read structure, walk
> the list of domains in this resource to find all that share an L3
> cache id (rr->display_id).

"L3 cache id" vs "monitor display scope"?

> 
> Adjust the RMID value based on which SNC domain is being accessed.

Thinking generously this changelog contains two brief descriptions of code
snippets. There is no context or explanation of what the goal of this change
is and why it chooses to implement things the way it does ... implementations
that definitely needs some explanation.

> 
> Signed-off-by: Tony Luck <tony.luck@intel.com>
> ---
>  arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 20 +++++++++++---
>  arch/x86/kernel/cpu/resctrl/monitor.c     | 33 ++++++++++++++++++++++-
>  2 files changed, 48 insertions(+), 5 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> index 3b9383612c35..7ab788d47ad3 100644
> --- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> +++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> @@ -575,15 +575,27 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
>  	resid = md.u.rid;
>  	domid = md.u.domid;
>  	evtid = md.u.evtid;
> -
> +	rr.sumdomains = md.u.sum;
>  	r = &rdt_resources_all[resid].r_resctrl;
> -	hdr = rdt_find_domain(&r->mon_domains, domid, NULL);
> -	if (!hdr || WARN_ON_ONCE(hdr->type != RESCTRL_MON_DOMAIN)) {
> +
> +	if (rr.sumdomains) {
> +		rr.display_id = domid;
> +		list_for_each_entry(d, &r->mon_domains, hdr.list) {
> +			if (d->display_id == domid)
> +				goto got_domain;
> +		}
>  		ret = -ENOENT;
>  		goto out;
> +	} else {
> +		hdr = rdt_find_domain(&r->mon_domains, domid, NULL);
> +		if (!hdr || WARN_ON_ONCE(hdr->type != RESCTRL_MON_DOMAIN)) {
> +			ret = -ENOENT;
> +			goto out;
> +		}
> +		d = container_of(hdr, struct rdt_mon_domain, hdr);
>  	}
> -	d = container_of(hdr, struct rdt_mon_domain, hdr);
>  
> +got_domain:
>  	mon_event_read(&rr, r, d, rdtgrp, evtid, false);

This looks more like "wedging things until it works". In the "sumdomains" case
it just picks the first matching domain without any explanation why it would
be ok. The only reason why mon_event_read() needs the domain is so that
it can use it to determine which CPUs it should read the counters from and it looks
like this code is written to take advantage of just that. It is a hack based on
knowledge of internals of mon_event_read() to just get the reader to run on a
CPU that works for SNC where another quirk awaits to override the domain
when the counter is _actually_ read so that it can get the correct architectural
state.

I was expecting here to at least see some documentation/comments
to explain why the code behaves the way it is. Optimistically it can
be documented as an "optimization" for the sumdomains case that is only
used by SNC where it is ok to read counters from any domain in the
"monitor display scope".

Apart from the documentation I do not think this code should wedge itself
in like this. 

To make it obvious what this code does I think the non-SNC case should
set rr.d and mon_event_read() should take a CPU mask as parameter
instead of a struct rdt_domain. This means that in SNC case rr.d will be
NULL and make the code flow more obvious.

>  
>  	if (rr.err == -EIO)
> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
> index 0f66825a1ac9..668d2fdf58cd 100644
> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
> @@ -546,7 +546,7 @@ static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid,
>  	}
>  }
>  
> -static int __mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr)
> +static int ___mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr)
>  {
>  	struct mbm_state *m;
>  	u64 tval = 0;
> @@ -569,6 +569,37 @@ static int __mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr)
>  	return 0;
>  }
>  
> +static u32 get_node_rmid(struct rdt_resource *r, struct rdt_mon_domain *d, u32 rmid)
> +{
> +	int cpu = cpumask_any(&d->hdr.cpu_mask);
> +
> +	return rmid + (cpu_to_node(cpu) % snc_nodes_per_l3_cache) * r->num_rmid;

Is this missing a "if (snc_nodes_per_l3_cache > 1)" ?

I do not think this belongs in resctrl fs code though. Should this algorithm be forced
on all architectures? It seems more appropriate for the arch specific code.

> +}
> +
> +static int __mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr)
> +{
> +	struct rdt_mon_domain *d;
> +	u32 node_rmid;
> +	int ret = 0;
> +
> +	if (!rr->sumdomains) {
> +		node_rmid = get_node_rmid(rr->r, rr->d, rmid);
> +		return ___mon_event_count(closid, node_rmid, rr);
> +	}
> +

/*
 * rr->sumdomains is only set by SNC mode where the event
 * counters of a monitoring domain can be read from a CPU belonging
 * to a different monitoring domain that shares the same monitoring
 * display domain. Optimize counter reads when needing to sum the
 * values by reading the counter for several domains from
 * the same CPU instead of sending IPI to one CPU of each monitoring
 * domain.
 */

Please feel free to improve, please do help folks trying to understand
what this code does.

> +	list_for_each_entry(d, &rr->r->mon_domains, hdr.list) {
> +		if (d->display_id != rr->display_id)
> +			continue;
> +		rr->d = d;
> +		node_rmid = get_node_rmid(rr->r, d, rmid);
> +		ret = ___mon_event_count(closid, node_rmid, rr);
> +		if (ret)
> +			break;
> +	}
> +
> +	return ret;
> +}
> +
>  /*
>   * mbm_bw_count() - Update bw count from values previously read by
>   *		    __mon_event_count().

Reinette