Received-SPF: pass (google.com: domain of linux-kernel+bounces-195883-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <9818c304-9056-4d79-acbe-2b35cb847ecd@intel.com>
Date: Thu, 30 May 2024 13:24:57 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v19 16/20] x86/resctrl: Make resctrl_arch_rmid_read()
 handle sum over domains
To: Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>, Maciej
 Wieczor-Retman <maciej.wieczor-retman@intel.com>, Peter Newman
	<peternewman@google.com>, James Morse <james.morse@arm.com>, Babu Moger
	<babu.moger@amd.com>, Drew Fustini <dfustini@baylibre.com>, Dave Martin
	<Dave.Martin@arm.com>
CC: <x86@kernel.org>, <linux-kernel@vger.kernel.org>,
	<patches@lists.linux.dev>
References: <20240528222006.58283-1-tony.luck@intel.com>
 <20240528222006.58283-17-tony.luck@intel.com>
From: Reinette Chatre <reinette.chatre@intel.com>
Content-Language: en-US
In-Reply-To: <20240528222006.58283-17-tony.luck@intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bDdtdUwvL2o0SUwxanpvdlREUmFjLzlvQklRY2cvQzRraGJqcktMaHNDbFIv?=
 =?utf-8?B?c2NRUFNtZkdWY3pnc2Q4UFpkSlZNNWdMQzcyS3IxenU5Lzhwd1NjRlBZdGlo?=
 =?utf-8?B?NGhJei8zZjcwOFBxTU1Wd1h0VE52OG9nWk9oWXNvOTR1ZHMrazVPSU9KYzgz?=
 =?utf-8?B?UUtHZ21RRlJiZ1dJZ2dMQzhML1Y4bWdjS21MK2xNSUh6bUMzWXluTUlPWWRj?=
 =?utf-8?B?Yi9za1dzK1dBYmJxRGVKNXB6Y1RuR3crbmUwejRKZm9GdTREaTRIL2ZjZ0Zu?=
 =?utf-8?B?ei9SUmpGZjVYTCtFbmxHNUw3aU0xblZVNGxWZjJYNDcvRENMVG5iWXhYYUJZ?=
 =?utf-8?B?SElrT25RUUxYZDJuWVdyQ2RPa0JpM2dPSTFaUjVCV2gxeFBHUkNHejZ3bnNR?=
 =?utf-8?B?bTlvTEtxekRmSXRtVnppQ25qUzUzUHIyMENRTlV4UVdMbS80UVBLYlY2ejBz?=
 =?utf-8?B?eCtlRkYvQ3Z1cFZZUWlKUEtsWlBEYXllR2hSL0RDM0ZML0Z1Y0tMZE84NTRO?=
 =?utf-8?B?cnpPdW5XTWlpUmNLU0FzZFR3R0dNbjhSNURUZzJYbyt0aUhRUTNxMGFsMytD?=
 =?utf-8?B?NTNJMVhiQW83YWxmalp4UFRmTHpZSEZJWGlsaXR4YjBwVlpVelZvVjRxZVpj?=
 =?utf-8?B?ZzZGYUk5RmlpWFZWWjNqTElYY21ua1E4Q09HamdXNysvT1l0TFpWNENiSGo5?=
 =?utf-8?B?WjVUZERQUzZid0pha0FucmNpbzJlQk5xclUwanFnTmRUUlIxbUlwWUV5TkEw?=
 =?utf-8?B?N3RHUWFHYWVHRkx4VUIxaG5ERlNOeWFCcmVUT1BaSVNjWk9UalAyKys4WWYv?=
 =?utf-8?B?enlYWGZOTE1IT1hVMlB1V0tZRW4xcUVXWHVESTQ4UUQ3TmpSWmo4OUpHR0xu?=
 =?utf-8?B?Q3gvM1hjcmpOK1ZjbmRhT0tINjVjd0VUTEJYUCtxNlVxbXFtL2M4VUk1VGxx?=
 =?utf-8?B?a2pjZU15ak1VLzFkaXVaUHZsSzgzWWdNZkxYRnJIb01TRkN1SjlWSWkwRGpR?=
 =?utf-8?B?R3JSWEhOZ21IM2RlQUtkb1dqdEhtMnBmMWw5MWxzWDFmdTVxVW1UZlVYSjl6?=
 =?utf-8?B?T0JXcXVFUUJsKzF2dFJPVGVpUHFNQXJhaks1MFZVWkxBTFJsL0s1aUdlWmhW?=
 =?utf-8?B?aXlsL01qVjJpK0lDOEpkOGtiNFBpODFpNWNQdDIrcDJSMXRKYmdORzRJMGU3?=
 =?utf-8?B?OVhIZHNmL2NTdSt1b2RnSGZTQ3VZdnhpWDNzYWxZazZCTUZTYUtLU2VPb0VF?=
 =?utf-8?B?QTBtZUJZS0Z2YUdaYnlZUTNFOGJCakc5dHk1dFlBaDFjV1dJTGpVVUM4cG5x?=
 =?utf-8?B?YXVRT0k2NkpvYm5XOUl2T3oyNmFWWVRaaGpKdjJ4Q3grOGJvVTZKQzFMUFlv?=
 =?utf-8?B?b3RvN1JGNFdTV0gvZnUyb1JTNlB2THdFR1hXL3RwdS85MlBYcTIvZ0NGbHgy?=
 =?utf-8?B?UkppeXB1Y3Z2STlzY1drbldQaDBqZzIyYnV1cnUwR1NvR2tFNEs4Vm8yOGZz?=
 =?utf-8?B?V3FLaGdrRlRrdytGZkJiaWFlV3lmQnBGU0J1OWJCc0JaanZhbTl2M21hUTFU?=
 =?utf-8?B?b2RFelRTM3lIOHo0Q0VndTl1Ump6MEdSbFZ4dWxCWkhHM0NVTzZKcndTQ1Iv?=
 =?utf-8?B?ZXcyWDJaNk1Qc2Z3aHRhcG9iMUJ5bVNkcUI4VXdqckNRa3haaGdycFBKWnM2?=
 =?utf-8?B?K0JPelgxT2xmUWxPdjZiWVJBdVR1Vm1HcUZsbGh0b0hrQ2NSSjd2cENLa2t5?=
 =?utf-8?B?M24xQXdGOE1HcEtCNGpRdVlxUHBiUEgvdysyM1UyQTZHMGd0dHR1cE9Wa3E0?=
 =?utf-8?B?bFcrZVlzMDJ3eVJadGduY3JzKzQ2RXZpdDZ1Z3RrZUpVMEYzVHd3dkxtcUxj?=
 =?utf-8?B?YXZzbzdCTGJKRnlYMmRncUhxQUozTUx1dnBUeERRVnNKdHFWMW9QTGU1aExY?=
 =?utf-8?B?THVZTUtIMGdRWUFqZ2xMK1N0REtjbnAzOWZJK2ZTeHYwanhqZ0dtbWYrWmp0?=
 =?utf-8?B?T1ZMam1Cd3RtcWhuK25Da1AyaHd5dm05WE5xU0hqWWE0MW1DZEJjVXUrTk92?=
 =?utf-8?B?WDlsaUtwZWxiZEhuSFJDb3JaaDMyN1VkNWwxbTB5MHlsQndNS0JGR2pIT2VY?=
 =?utf-8?B?N0xWY3dnWDVSRS8rclFxRHdHYWRmeFdweGZpUnV1RmRXeFlPQytlZ3VOY29S?=
 =?utf-8?B?aVE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b5e1e514-bc63-4d4e-7bec-08dc80e69461
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2024 20:25:00.1126
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1pK1YgnHMFiG4yIIeWWOqUj6hAWGK/dhuTpiPtqbaVlRlzWxMUo/fljEQ9IdeOJ2Twu5YPkylOzB0mnai8ckl77mbGr9KzmURUccbd/jNXo=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB6772
X-OriginatorOrg: intel.com

Hi Tony,

On 5/28/24 3:20 PM, Tony Luck wrote:
> Legacy resctrl monitor files must provide the sum of event values across
> all Sub-NUMA Cluster (SNC) domains that share an L3 cache instance.
> 
> Rename the existing resctrl_arch_rmid_read() function as
> resctrl_arch_rmid_read_one() (with some small refactoring to drop
> arguments that are not needed.

Missing closing ")".

> 
> Create a new resctrl_arch_rmid_read() that iterates across
> domains when necessary. Pass a CPU number from the right domain to
> resctrl_arch_rmid_read_one().

"when necessary"? Can you elaborate?
"Pass a CPU ..." that just describes code that can be learned from patch.
Please describe the changes not the code.

> 
> Signed-off-by: Tony Luck <tony.luck@intel.com>
> ---
>   arch/x86/kernel/cpu/resctrl/monitor.c | 41 ++++++++++++++++++++-------
>   1 file changed, 31 insertions(+), 10 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
> index 5f89ed4823ee..c9dd6ec68fcd 100644
> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
> @@ -200,10 +200,9 @@ static inline struct rmid_entry *__rmid_entry(u32 idx)
>    * Caller is responsible to make sure execution running on a CPU in
>    * the domain to be read.
>    */
> -static int logical_rmid_to_physical_rmid(int lrmid)
> +static int logical_rmid_to_physical_rmid(int cpu, int lrmid)
>   {
>   	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
> -	int cpu = smp_processor_id();
>   
>   	if (snc_nodes_per_l3_cache  == 1)
>   		return lrmid;
> @@ -211,13 +210,13 @@ static int logical_rmid_to_physical_rmid(int lrmid)
>   	return lrmid + (cpu_to_node(cpu) % snc_nodes_per_l3_cache) * r->num_rmid;
>   }
>   
> -static int __rmid_read(u32 lrmid,
> +static int __rmid_read(int cpu, u32 lrmid,
>   		       enum resctrl_event_id eventid, u64 *val)
>   {
>   	u64 msr_val;
>   	int prmid;
>   
> -	prmid = logical_rmid_to_physical_rmid(lrmid);
> +	prmid = logical_rmid_to_physical_rmid(cpu, lrmid);
>   	/*

Passing CPU as parameter to __rmid_read(), which is run via IPI, really
obfuscates the code. How about renaming it to "__rmid_read_phys()"
and provide it the "physical RMID" as parameter to make it clear what it is
doing?
That would mean an extra call to determine "physical RMID" before calling
__rmid_read_phys() but making it clear that it needs a physical RMID should
help to explain what is going on.


>   	 * As per the SDM, when IA32_QM_EVTSEL.EvtID (bits 7:0) is configured
>   	 * with a valid event code for supported resource type and the bits
> @@ -269,7 +268,7 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
>   		memset(am, 0, sizeof(*am));
>   
>   		/* Record any initial, non-zero count value. */
> -		__rmid_read(rmid, eventid, &am->prev_msr);
> +		__rmid_read(smp_processor_id(), rmid, eventid, &am->prev_msr);
>   	}
>   }
>   
> @@ -298,9 +297,8 @@ static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr, unsigned int width)
>   	return chunks >> shift;
>   }
>   
> -int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
> -			   u32 unused, u32 rmid, enum resctrl_event_id eventid,
> -			   u64 *val, bool sum, struct cacheinfo *ci, void *ignored)
> +static int resctrl_arch_rmid_read_one(struct rdt_resource *r, struct rdt_mon_domain *d,
> +				      int cpu, u32 rmid, enum resctrl_event_id eventid, u64 *val)
>   {
>   	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
>   	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
> @@ -313,7 +311,7 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
>   	if (!cpumask_test_cpu(smp_processor_id(), &d->hdr.cpu_mask))
>   		return -EINVAL;
>   
> -	ret = __rmid_read(rmid, eventid, &msr_val);
> +	ret = __rmid_read(cpu, rmid, eventid, &msr_val);
>   	if (ret)
>   		return ret;
>   
> @@ -327,7 +325,30 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
>   		chunks = msr_val;
>   	}
>   
> -	*val = chunks * hw_res->mon_scale;
> +	*val += chunks * hw_res->mon_scale;

The various new layers of indirection with SNC logic scattered between them
makes this change hard to understand.

> +
> +	return 0;
> +}
> +
> +int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
> +			   u32 unused, u32 rmid, enum resctrl_event_id eventid,
> +			   u64 *val, bool sum, struct cacheinfo *ci, void *ignored)

This is not architecture specific code.

> +{
> +	int cpu = smp_processor_id();
> +	int ret;
> +
> +	*val = 0;
> +	if (!sum)
> +		return resctrl_arch_rmid_read_one(r, d, cpu, rmid, eventid, val);
> +

	/* SNC quirk that needs to be documented */

> +	list_for_each_entry(d, &r->mon_domains, hdr.list) {
> +		if (d->ci->id != ci->id)
> +			continue;
> +		cpu = cpumask_any(&d->hdr.cpu_mask);
> +		ret = resctrl_arch_rmid_read_one(r, d, cpu, rmid, eventid, val);

The cpu parameter can be dropped, no? With the domain provided to resctrl_arch_rmid_read_one()
it is not necessary to again split the SNC logic (in this case the "reading from any CPU
in the cache domain is ok but still need accurate arch state") across multiple layers,
just contain this in (documented portion of) resctrl_arch_rmid_read_one().

> +		if (ret)
> +			return ret;
> +	}
>   
>   	return 0;
>   }

Reinette