Received-SPF: pass (google.com: domain of linux-kernel+bounces-145678-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <0698418a-aede-40d6-90ce-dbf6e8796916@intel.com>
Date: Mon, 15 Apr 2024 10:47:55 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v1 05/31] x86/resctrl: Remove rdtgroup from
 update_cpu_closid_rmid()
To: Dave Martin <Dave.Martin@arm.com>
CC: James Morse <james.morse@arm.com>, <x86@kernel.org>,
	<linux-kernel@vger.kernel.org>, Fenghua Yu <fenghua.yu@intel.com>, "Thomas
 Gleixner" <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>, "Borislav
 Petkov" <bp@alien8.de>, H Peter Anvin <hpa@zytor.com>, Babu Moger
	<Babu.Moger@amd.com>, <shameerali.kolothum.thodi@huawei.com>, "D Scott
 Phillips OS" <scott@os.amperecomputing.com>, <carl@os.amperecomputing.com>,
	<lcherian@marvell.com>, <bobo.shaobowang@huawei.com>,
	<tan.shaopeng@fujitsu.com>, <baolin.wang@linux.alibaba.com>, Jamie Iles
	<quic_jiles@quicinc.com>, Xin Hao <xhao@linux.alibaba.com>,
	<peternewman@google.com>, <dfustini@baylibre.com>, <amitsinght@marvell.com>,
	David Hildenbrand <david@redhat.com>, Rex Nie <rex.nie@jaguarmicro.com>
References: <20240321165106.31602-1-james.morse@arm.com>
 <20240321165106.31602-6-james.morse@arm.com>
 <6bbff669-cbe5-4284-b64a-4825a541b35f@intel.com>
 <ZhldWJVck7VmU3G3@e133380.arm.com>
Content-Language: en-US
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <ZhldWJVck7VmU3G3@e133380.arm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?alJ0dHNuVmVEbTdkbHAxNXdXZHA3SkVMYkpXMUNNejd2SWNIUDVaV0FFdGkx?=
 =?utf-8?B?K0tNbHBBRjJUUThxNmphbDZWT0tsTFVLWDFjN3JoS0VCR1dYSVlkbmRYQjlV?=
 =?utf-8?B?cDZtdkFjZnBMQWlKSHliWU0zMkIvVThpaWR5UjhPSDY5WEo1NW0yQm0xRHN3?=
 =?utf-8?B?RXpGVFNZUFg0MGgrZ0JhWWJKRk1mSnVIVE5JUmMwUDYyNmpUY2hiVTk4V0pU?=
 =?utf-8?B?cnpCU3kyTkczYnVDbXVnYlVBc2s4TmQ0SGhPL0QvaHR6OGFWclVaRHltQzJv?=
 =?utf-8?B?SENXOVNaYnh6dnU1dE1wRnRKcFhJdDNaOHNPWGlBNEdZcFRZMmlUSzl0YkxU?=
 =?utf-8?B?L1UxNnl6TUlUMlA0R0pTSUFoeTZzb2gxb2ZaeDRvTEgwOW44SFpKQ2k3akVi?=
 =?utf-8?B?eVl2eE9IYXRaUzk4bGtnUHp3dzV2cjFKTkorNHc1M3d4WU9YaDJQOWQxeGxr?=
 =?utf-8?B?M2dFVEZrQzZHTytPZmxRM3ZKeTJhdktXcDBPd25VSlIvWGpPZ1FqNmYzWUdX?=
 =?utf-8?B?Y2ZaNVJvYW9GTkwyS3doNkt2WFptaTVZbzNvbnpDU0xvcGZSSDBNb2xVZjZs?=
 =?utf-8?B?TmIrK2FXVDQvOFJPRjhyZTI1WnduMzRkVWo1UEJ6TEdmb3JlbHlTNVRySkdP?=
 =?utf-8?B?UEFsWEdhOHFTdUUzKzJlcEJnSFFCbDExeEdjQXZ4eDBtMEVPczA5bXBUSTJ2?=
 =?utf-8?B?UXpwR3QzMXNKelpVVzRidUJpZGNXZG82WWxjUWY0WGVaSFJKcnp6Y3owQ09Y?=
 =?utf-8?B?WEVBeGFBQU9YTkhyVlBaYUwraDlhV3ZwWTdlcWVtb1lXL09aNGo3MGJhN2xs?=
 =?utf-8?B?TmdNVUM0RENDUWdzdVpuMGhTOGNYK2RsSEJmaCtmMEJRYWNTR2tSdkRVSVRE?=
 =?utf-8?B?YUg3TXpldW94Mk00Q05ld0VjN0JrKzRwN3hxWDRYNjNIbThzZ3VZWlJ3Mlc2?=
 =?utf-8?B?UHlZOHZPZ0NVZjJTdGkvZmtoVWhCa0xxNWtmbEVmM1NtSE8wUmN5RkhLMUlp?=
 =?utf-8?B?TkdnS2pGU1oyTzQwbTk3RzFRVkMrM3ZKa3IzRDhUTEZZWEs2T0pnWi9YS0JZ?=
 =?utf-8?B?ZnBwcVViTXl1MmZ2U3NIbkZjKzdGZlhBSE50b1dLakVsenpCb0t0bTZpZ3cx?=
 =?utf-8?B?VDFDMHczVSs2RWQxNTg2RVBWbXpSaW5FeW8yVlgzaVBBY0tTRW1oc0lZY2JY?=
 =?utf-8?B?UUE5OUJxY3VPUzRtTmp4M3NhQnFiQWthVUNpVG9TTjRIaWVxTzRhTGFjUHpQ?=
 =?utf-8?B?cjlwODFlVzVBR3ExNC9QQVNRRnpLb3Z4K2R3ZktpU2RHL3ZJOE9YK3UrNnhi?=
 =?utf-8?B?WkxPeTBKK2VnNXVCTnlqMmN1NUhGdGtUaEZLVGxkaThhTVczeEFNQjFVYlpJ?=
 =?utf-8?B?UE9nQzBwN2dZcDhQMWJrN0VoRGhuRVVpaXBWenV6bG5Jd1gvUTVKa0hzK1Ji?=
 =?utf-8?B?WW50UjlhRUEzTGFwZVNOV3RhUVpwYWJpM3ZEWjNNZHNBVUNTYU4rTEw1MGc3?=
 =?utf-8?B?T0ErQlpvSlRjTFJmOTZMTEZ4TkdZV2hJR1FNS3d0cW8xT08rSnk4a3AyOHQ2?=
 =?utf-8?B?UTBMbFNnekNWZlowN0Z5NUJDeWROa044SlRuSDlRMk4zZkVFOVNScVlYSEpl?=
 =?utf-8?B?YzRyTUhJZGFtM1EyMHI2UmxGQzMwK0NjVzZkTXVwSlBLZlMzOUxodmRRTkJa?=
 =?utf-8?B?VFUyWjIyVkJBWFU3V0lrSjJYTDQ3ckg1cVRsNm5VU2Q0Wk1oc2hrSlhsTExM?=
 =?utf-8?B?MzNRTm94b0RnNnJaYTVuOXBnTzltRmZxTmFUTlY3dlFkOVgvN0ttZ0V0bzlZ?=
 =?utf-8?B?cWlDSlVjaUFOMldqZThsQUZhdG5pSkp2akVOWW1jVE9yNlVBL0J6a1kraGx5?=
 =?utf-8?B?T2lOR0REZHJCb0FrbE90WHhDc1FyUkJJZGYvdnA4b0lHeFVTZ2xOTWxvMUVw?=
 =?utf-8?B?R1JIdzNWS3RKWi9WQjkwaSt6K1RYS2NXZjUvRHRQR2E4T1QvMUEvMk1JeWk0?=
 =?utf-8?B?eG5nTkxRdVVYUEVTbzRiMExSY1A3aytNODMrV3BsS0VVTmVHaWxNRzBzKy9h?=
 =?utf-8?B?U1gyaC9XNU1ibW9tQTFiWUNBUm9YYWxHRDFnclN2eGpRVEx1OUJMYkdBazZw?=
 =?utf-8?B?NXk3T2RjNGUxNk1pNFVCKzU4RHl2MUdjMnZPZk9FS3B6dC80c3ptT3BLMVA1?=
 =?utf-8?B?SlE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 8ad78344-1947-4942-331c-08dc5d743011
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Apr 2024 17:47:58.5818
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: zP58Z8h9fTNbeNpAQTCbt6IsGptELxR7Jwpn+nvdTziXaMuwZIokqzVPBf9CcKxOdG6i6WXHiHRVRI1F7R4M6ALcdC4VpjziVhyISY8kSm4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB6751
X-OriginatorOrg: intel.com

Hi Dave,

On 4/12/2024 9:12 AM, Dave Martin wrote:
> On Mon, Apr 08, 2024 at 08:16:08PM -0700, Reinette Chatre wrote:
>> Hi James,
>>
>> On 3/21/2024 9:50 AM, James Morse wrote:
>>> update_cpu_closid_rmid() takes a struct rdtgroup as an argument, which
>>> it uses to update the local CPUs default pqr values. This is a problem
>>> once the resctrl parts move out to /fs/, as the arch code cannot
>>> poke around inside struct rdtgroup.
>>>
>>> Rename update_cpu_closid_rmid() as resctrl_arch_sync_cpus_defaults()
>>> to be used as the target of an IPI, and pass the effective CLOSID
>>> and RMID in a new struct.
>>>
>>> Signed-off-by: James Morse <james.morse@arm.com>
>>> ---
>>>  arch/x86/kernel/cpu/resctrl/rdtgroup.c | 19 +++++++++++++++----
>>>  include/linux/resctrl.h                | 11 +++++++++++
>>>  2 files changed, 26 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>>> index 5d2c1ce5b6b1..18f097fce51e 100644
>>> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>>> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>>> @@ -341,13 +341,13 @@ static int rdtgroup_cpus_show(struct kernfs_open_file *of,
>>>   * from update_closid_rmid() is protected against __switch_to() because
>>>   * preemption is disabled.
>>>   */
>>> -static void update_cpu_closid_rmid(void *info)
>>> +void resctrl_arch_sync_cpu_defaults(void *info)
>>>  {
>>> -	struct rdtgroup *r = info;
>>> +	struct resctrl_cpu_sync *r = info;
>>>  
>>>  	if (r) {
>>>  		this_cpu_write(pqr_state.default_closid, r->closid);
>>> -		this_cpu_write(pqr_state.default_rmid, r->mon.rmid);
>>> +		this_cpu_write(pqr_state.default_rmid, r->rmid);
>>>  	}
>>>  
>>>  	/*
>>> @@ -362,11 +362,22 @@ static void update_cpu_closid_rmid(void *info)
>>>   * Update the PGR_ASSOC MSR on all cpus in @cpu_mask,
>>>   *
>>>   * Per task closids/rmids must have been set up before calling this function.
>>> + * @r may be NULL.
>>>   */
>>>  static void
>>>  update_closid_rmid(const struct cpumask *cpu_mask, struct rdtgroup *r)
>>>  {
>>> -	on_each_cpu_mask(cpu_mask, update_cpu_closid_rmid, r, 1);
>>> +	struct resctrl_cpu_sync defaults;
>>> +	struct resctrl_cpu_sync *defaults_p = NULL;
>>
>> Please maintain reverse fir order.
> 
> Or, more tersely as follows?
> 
> 	struct resctrl_cpu_sync defaults, *defaults_p = NULL;

Sure.

> 
> "Reverse fir order" seems to be documented as a preference rather than a
> rule.

This does not seem to be a place that warrants an exception to this
preference. Note how this function is not consistent with any other
in the file.

> The declarations can be swapped, but defaults_p is in some sense a weak
> pointer to defaults, so it feels a bit strange to declare them backwards.
> 
> Alternatively, could we rename defaults_p to p?  Given the size of this
> function I don't think that impacts clarity.

Do you imply that this would maintain the order in this patch? It does
not look to me that it would but I may be looking wrong.

sidenote: the "on_each_cpu_mask()" in update_closid_rmid() can be on
one line.

> 
> I'll wait for your opinion on this.
> 
> 

..

>>> + * struct resctrl_cpu_sync, or NULL.
>>> + */
>>
>> Updating the CPU's defaults is not the primary goal of this function and because
>> of that I do not think this should be the focus with the main goal (updating
>> RMID and CLOSID on CPU) ignored. Specifically, this function only updates
>> the defaults if *info is set but it _always_ ensures CPU is running with
>> appropriate CLOSID/RMID (which may or may not be from a CPU default).
>>
>> I think resctrl_arch_sync_cpu_closid_rmid() may be more appropriate
>> and the comment needs to elaborate what the function does.
>>
>>> +void resctrl_arch_sync_cpu_defaults(void *info);
> 
> That seems reasonable, and follows the original naming and what the
> code does:
> 
> What about:
> 
> /**
>  * resctrl_arch_sync_cpu_defaults() - Refresh the CPU's CLOSID and RMID.
>  *				      Call via IPI.

Did you intend to change function name?

How about "Refresh the CPU's ..." -> "Refresh this CPU's ..." I think it
makes it more obvious how this function is called.

>  * @info:	If non-NULL, a pointer to a struct resctrl_cpu_sync specifying
>  *		the new CLOSID and RMID for tasks in the default resctrl ctrl
>  *		and mon group when running on this CPU.  If NULL, the default
>  *		CLOSID and RMID are not changed.

"If NULL, this CPU is not re-assigned to a different group." ?

>  *
>  * This is how reassignment of CPUs and/or tasks to different resctrl groups
>  * is propagated when requested by the resctrl fs core code.

Could you please use imperative tone here?  For example, "Propagates reassignment
of CPUs and/or tasks to different resctrl groups."

>  *
>  * This function should typically record the per-cpu defaults specified by

"should" sounds like there may be cases when this is not done? Maybe just
"Records the per-CPU defaults specified ..."

>  * @info (if any), and then reconfigure the CPU's hardware CLOSID and RMID
>  * for subsequent execution based on @current, in the same way as during a
>  * task switch.
>  */
> 
> ...?

Reinette