Received-SPF: pass (google.com: domain of linux-kernel+bounces-136183-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <6bbff669-cbe5-4284-b64a-4825a541b35f@intel.com>
Date: Mon, 8 Apr 2024 20:16:08 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v1 05/31] x86/resctrl: Remove rdtgroup from
 update_cpu_closid_rmid()
To: James Morse <james.morse@arm.com>, <x86@kernel.org>,
	<linux-kernel@vger.kernel.org>
CC: Fenghua Yu <fenghua.yu@intel.com>, Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>, H Peter Anvin
	<hpa@zytor.com>, Babu Moger <Babu.Moger@amd.com>,
	<shameerali.kolothum.thodi@huawei.com>, D Scott Phillips OS
	<scott@os.amperecomputing.com>, <carl@os.amperecomputing.com>,
	<lcherian@marvell.com>, <bobo.shaobowang@huawei.com>,
	<tan.shaopeng@fujitsu.com>, <baolin.wang@linux.alibaba.com>, Jamie Iles
	<quic_jiles@quicinc.com>, Xin Hao <xhao@linux.alibaba.com>,
	<peternewman@google.com>, <dfustini@baylibre.com>, <amitsinght@marvell.com>,
	David Hildenbrand <david@redhat.com>, Rex Nie <rex.nie@jaguarmicro.com>, Dave
 Martin <dave.martin@arm.com>
References: <20240321165106.31602-1-james.morse@arm.com>
 <20240321165106.31602-6-james.morse@arm.com>
Content-Language: en-US
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20240321165106.31602-6-james.morse@arm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eCt4WVhONFRmZzhnTVkzc1VyU0NtMzNJWjIwVlR1bzROTk5rU0hzbS9TcTNV?=
 =?utf-8?B?QUtSZUhWNW0rdlZYMWJuOHlack4wcXZtNHYydlZIZDFSaHdvMFNRNEJOUlVV?=
 =?utf-8?B?dzVDOHNKNTR2d2JlRVVxQThodmkybTlIOU5YYlMyTFp1aXRYc0lIdHJIaW9O?=
 =?utf-8?B?VDBXU283ZnlodGl3a29wbW9QUTA1bS8wUW4zR1Y0endwR2FEcUg4ZXRiNzJM?=
 =?utf-8?B?UXNYRWlkVTk1NENXVi9Da3pscXNkWmlPcnU3dzRXSDVVRXpkdjczVUtBdDE2?=
 =?utf-8?B?TGZ0V3BPOXg3YmNuVFFGcGFPTHJPZHpLV3hWRG1ZY0VNQnlENzNnNGZ2WFhy?=
 =?utf-8?B?cm0xNmR2NlZzMlJxK044aFh1SnlNY09hQlhKRFBFSDFvRnVZcXRhYTlpeEw5?=
 =?utf-8?B?TlI2bmNvWTgzcUcza1dIOVBDc0NOdjhrMm5jMGQ0NkxVeHRLcnBvNUQ0Z3Jv?=
 =?utf-8?B?cStVN0hGclpRcU4wTzE2NUJMMUVqNzA2anhXMmZGcTgreUZQNnBnWThxRER2?=
 =?utf-8?B?Z2ozTjBZTXpYeVVGMFVZKy9LWWtQaFhsejZyT043V3dwc0pPOENkbmxMdXp6?=
 =?utf-8?B?MHJzVDhkbWJjQkRibXJHaHJnOHZBeXRodEdzTFNwTVQwcnFOeVluTjZrR2dL?=
 =?utf-8?B?Y1U2aTFaWTNlK1QyNElWa1B5MmRSZmR1NTZ4N0Q4aEdjTlZTNVMvN1pvazlk?=
 =?utf-8?B?SE5qN09CdkRlVksrNUlZYjkyQWNOamtLSXJYa0NqUTU1Sk1MYlgvTjBVWVBY?=
 =?utf-8?B?aEN4ekxucllaNWJlazkzMjJzems4QzhSbVhwckZ4eC8yTFJtaWN4d2dkbndx?=
 =?utf-8?B?Q09hTnNSVHlicnZIQ1BXVnJDdXgzSkZWczdYUUtVWGNvMWRtR1ZQbEF5R3lx?=
 =?utf-8?B?QUJqOVllby93Qm9TaWJxRmdSUHBqOXB5ek4rVzBJYk5saHRRNFZVQkludTA4?=
 =?utf-8?B?QUxNMDE2RVd3cWZBbFRLSENPZGNNSlRGVDBsbHRyQmphQWFqNTdGTUFRbWo4?=
 =?utf-8?B?YnZicmN3K3pmcmVwaGQ2L1B3NUNneHQrLyswTXZxZGdiWFJUWXNMK1RXM3ht?=
 =?utf-8?B?WXFnS2tjQ3I5Y2tIOHpodks3MktybSszN0xYUExicGF5dGpjai8wd01IODZZ?=
 =?utf-8?B?RE9KRysxOEltNUZYMmFTSUVpR3l1QVRIUlp5blBZYmwwQkVQY0VWU05zWVc0?=
 =?utf-8?B?aTM1TUZ3bzRJMVo0SE4wYk1SZkhJMjExWi95TDV1UzVDaU42ajg5N3RnR3NW?=
 =?utf-8?B?SWMwZHlzbEFTUDV0a3EybVVpK1hqSnhWZU5MYVhCc2hxZmZvNGVKaG0wdGRV?=
 =?utf-8?B?c25NNmo4WFpxL0h1NVhRenVYd0c4RXQ5Y3EzcHhITXA4bnIwSzZxcWpIVEpG?=
 =?utf-8?B?YnNsNDM0TUdaM09vUklGT2MrdU8ycVlEYVBIOXhyVWN1WjZ0OEQ1aWNkd2hP?=
 =?utf-8?B?NDlZSm5WcnlTYUpTa0FNaHZ6ZTVpWkF5bWx3UmpLSlBTaVRxeXNrck9JTDBx?=
 =?utf-8?B?S3ZKNURQc25hc05nQjJ3TjJUUHdXWXJzMW1RZWJNVzRiSlBmQlN3Zk9TRFd0?=
 =?utf-8?B?WEJGaDhFNUJMWnJxemczczkvSklnVjhRRXhYM3lmTUp2K0I2enoyYTdqWHBt?=
 =?utf-8?B?ZDFuTVB2VVptNUl1VWZBc01Vb0wrZEcwOFNCb0pidUhMcFRxWW9DZVdZYUsx?=
 =?utf-8?B?ZU9ySEI1TkEzTUFSQS9uWi9XR01LVXo0N2p4MCswMXNocjkreFlnanNxMDlV?=
 =?utf-8?B?UE9OQ0N3SE9SQ3BNaXg0QmwxeDZqQitIb2V6cENNdFY1dFNNMDhsWjZ4aUlw?=
 =?utf-8?B?QXJNWkNIRUEyY1Q0bkt6SlBleFJVYXd5MHUvem10Q2RGVnY0c3JRUm4ramZh?=
 =?utf-8?B?NU9QcnBnNW15WDFOc0szcU0wN2FyVGcrT0I2bmVwamxUZEd4Tkg0V1BqSHZ2?=
 =?utf-8?B?aVVRd2o2dTQ1WmlGU2ZKYnJUSm9PTE9uaUh1Nnc3WEpRQy95TjRMaDBaWUZz?=
 =?utf-8?B?TitxYlFIOENROVJTLzREYlM4akpQNmRFdGNaVEdXcUlHemliNFdXSmRDKzNQ?=
 =?utf-8?B?SXVoOHduaXl5ejJDL3VmRTBNOTM2VWI5eGRiM29neTVhUFhpcStneTdCcG13?=
 =?utf-8?B?WW1mZDVlMWRwUUkzWTN6MGVwZnNGOXlXWHk3VDhrMk1tLzVVNHZKOXhQSlRx?=
 =?utf-8?B?MWc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 20b64b79-31d0-4828-58f9-08dc5843680e
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Apr 2024 03:16:11.2616
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 7s4WT+3KU6JgMteDf4p+MTMZrkK8Ka8vuoL5pZdgEKxWL2vB/lin0dcmpyagaqOQ0if/H0OaC4KL/MVQH22sVX7G8t3E+K+5fDksreY7L84=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB4783
X-OriginatorOrg: intel.com

Hi James,

On 3/21/2024 9:50 AM, James Morse wrote:
> update_cpu_closid_rmid() takes a struct rdtgroup as an argument, which
> it uses to update the local CPUs default pqr values. This is a problem
> once the resctrl parts move out to /fs/, as the arch code cannot
> poke around inside struct rdtgroup.
> 
> Rename update_cpu_closid_rmid() as resctrl_arch_sync_cpus_defaults()
> to be used as the target of an IPI, and pass the effective CLOSID
> and RMID in a new struct.
> 
> Signed-off-by: James Morse <james.morse@arm.com>
> ---
>  arch/x86/kernel/cpu/resctrl/rdtgroup.c | 19 +++++++++++++++----
>  include/linux/resctrl.h                | 11 +++++++++++
>  2 files changed, 26 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> index 5d2c1ce5b6b1..18f097fce51e 100644
> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> @@ -341,13 +341,13 @@ static int rdtgroup_cpus_show(struct kernfs_open_file *of,
>   * from update_closid_rmid() is protected against __switch_to() because
>   * preemption is disabled.
>   */
> -static void update_cpu_closid_rmid(void *info)
> +void resctrl_arch_sync_cpu_defaults(void *info)
>  {
> -	struct rdtgroup *r = info;
> +	struct resctrl_cpu_sync *r = info;
>  
>  	if (r) {
>  		this_cpu_write(pqr_state.default_closid, r->closid);
> -		this_cpu_write(pqr_state.default_rmid, r->mon.rmid);
> +		this_cpu_write(pqr_state.default_rmid, r->rmid);
>  	}
>  
>  	/*
> @@ -362,11 +362,22 @@ static void update_cpu_closid_rmid(void *info)
>   * Update the PGR_ASSOC MSR on all cpus in @cpu_mask,
>   *
>   * Per task closids/rmids must have been set up before calling this function.
> + * @r may be NULL.
>   */
>  static void
>  update_closid_rmid(const struct cpumask *cpu_mask, struct rdtgroup *r)
>  {
> -	on_each_cpu_mask(cpu_mask, update_cpu_closid_rmid, r, 1);
> +	struct resctrl_cpu_sync defaults;
> +	struct resctrl_cpu_sync *defaults_p = NULL;

Please maintain reverse fir order.

> +
> +	if (r) {
> +		defaults.closid = r->closid;
> +		defaults.rmid = r->mon.rmid;
> +		defaults_p = &defaults;
> +	}
> +
> +	on_each_cpu_mask(cpu_mask, resctrl_arch_sync_cpu_defaults, defaults_p,
> +			 1);
>  }
>  
>  static int cpus_mon_write(struct rdtgroup *rdtgrp, cpumask_var_t newmask,
> diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
> index 6e87bc95f5ea..2b79e4159507 100644
> --- a/include/linux/resctrl.h
> +++ b/include/linux/resctrl.h
> @@ -220,6 +220,17 @@ struct resctrl_schema {
>  	u32				num_closid;
>  };
>  
> +struct resctrl_cpu_sync {
> +	u32 closid;
> +	u32 rmid;
> +};
> +
> +/*
> + * Update and re-load this CPUs defaults. Called via IPI, takes a pointer to

"this CPU's defaults"?

> + * struct resctrl_cpu_sync, or NULL.
> + */

Updating the CPU's defaults is not the primary goal of this function and because
of that I do not think this should be the focus with the main goal (updating
RMID and CLOSID on CPU) ignored. Specifically, this function only updates
the defaults if *info is set but it _always_ ensures CPU is running with
appropriate CLOSID/RMID (which may or may not be from a CPU default).

I think resctrl_arch_sync_cpu_closid_rmid() may be more appropriate
and the comment needs to elaborate what the function does.

> +void resctrl_arch_sync_cpu_defaults(void *info);
> +
>  /* The number of closid supported by this resource regardless of CDP */
>  u32 resctrl_arch_get_num_closid(struct rdt_resource *r);
>  int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid);

Reinette