Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <85e9d499-5393-f388-7d23-aa68248e5037@intel.com>
Date:   Mon, 21 Nov 2022 13:59:15 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.5.0
Subject: Re: [PATCH v3 2/2] x86/resctrl: update task closid/rmid with
 task_call_func()
Content-Language: en-US
To:     Peter Newman <peternewman@google.com>, <fenghua.yu@intel.com>
CC:     <bp@alien8.de>, <derkling@google.com>, <eranian@google.com>,
        <hpa@zytor.com>, <james.morse@arm.com>, <jannh@google.com>,
        <kpsingh@google.com>, <linux-kernel@vger.kernel.org>,
        <mingo@redhat.com>, <tglx@linutronix.de>, <x86@kernel.org>
References: <20221115141953.816851-1-peternewman@google.com>
 <20221115141953.816851-3-peternewman@google.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20221115141953.816851-3-peternewman@google.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MzRXbGtTcGtOUFFrUnpJUWlZL2lya3FSTU9jUmU2bW1oZHJYdzczL0wyMjhX?=
 =?utf-8?B?V0MvUXNwSVUrN3JtQ1duRTRvSXlnUXhxcHh4TU9zZ29zekwrWnpWbFhTK3JR?=
 =?utf-8?B?QzRCTUFTNlFkNitKSGYvWEg0ekRKWU5kbzBBTk1sYkFOSkJoNkxHUm5mNUhU?=
 =?utf-8?B?TVBnemNpY1NrVy9YcXQrdThmRjBWYnVEZmUxamNPOFdPcjRhVGVXMjJpS0F5?=
 =?utf-8?B?bEpRbUxYZnpMb0hPeEwrS3pCNzNTMHFCK0FWNjNHSHZ2Z2hlUTBKcWVBOGRO?=
 =?utf-8?B?dzJLY09Kb2tYVzlWTkRpVzBVUHRTdFJTQzBBUExpZXFsNVVhK3hWVVE2S2c0?=
 =?utf-8?B?QThtWFQwcUhLd2FMcDJxc0d3ZmJCUnhZR0dCaHlSYnhOVlcvTloyRWVCSit5?=
 =?utf-8?B?QWtyLy8rWDNDcjJWRG43SHpFSW1GekFoT1VHb0RUNWs2RzBBdFRZUExKd2NL?=
 =?utf-8?B?VnJTek9DWGkwYm93KzdDTzVwQmhRcFg0RzFzMVRrTzAwaFB1TkRlMno4dENr?=
 =?utf-8?B?akM0ZHJEVUVHTS9yTGx4MkRITkJrYkZDNzV2ZENkdWIyTURDc3Q5cXc2QWt1?=
 =?utf-8?B?RzJ3RmxLcS9VUm1qa0VOWmpjYkppYXV2WGpkendqYTBuM1hiZG9YaEl3clg4?=
 =?utf-8?B?QlhyMllaMFFWMjdMRER1aG1kMEkxNnJKc3NrdFZXcXZxaU9CdVdJL2ovMGxM?=
 =?utf-8?B?QSt6OG05SnRnNXIwT0hWWkwwN3ZjWGtVcDR5N21RbHNSamxqNU9nSnlWaWhS?=
 =?utf-8?B?MnR3eWsxM2hiTElqRkM3NmdycGtVakxQMmxwcTBwYkE2ampLK0tpUjRuaDVo?=
 =?utf-8?B?TkNQREtDMWJVMWpPRkt3U3RKUGg0eWJYa3VPOUd4UFJ0SkJDMk1lUytZTE1u?=
 =?utf-8?B?VVJORXZsenFabEJjeU5HV25WbXBZZDdFYWRaZGovS0daMW45b3REaW9FME50?=
 =?utf-8?B?UXJvcTZTOVVZNUQ5M1AyeXR6SFlYS2U1RUJQOFRBNkl0MDB3eHZ4ODkvU1Mz?=
 =?utf-8?B?WktxYTlJN1FtRW5FK2hPUG5aMmw2ZWxwRGs2bHJwcFNySUVTQVdJZDZhQU9L?=
 =?utf-8?B?azVoNzhIVjVvNEpnVHdsUFhWSUNIM0FuN3Q0U2RNbm1weTJOSXhSWW9vTXpk?=
 =?utf-8?B?Y2l6M0RMV3RmMnBXT1FuMEZBOUh1eVFKa2pPdEx0KzM2YjV1MmwwWnArMk83?=
 =?utf-8?B?R0V1Z3VRN3lySVJaSE1KYnpmMDJIcllZTXg4MnRva2tYb2gxL0pDL1Y2NWhZ?=
 =?utf-8?B?c1J1RXFXMHJrdFgvc3hCTkhjVmxyYlhFQmp1ZFE3Qkc5V0lkdDhKcklxSVdt?=
 =?utf-8?B?MTZVb2tOTEsxcDBsOGlUNjFyeWprbEN2Ry9pcXRjdjkrL0lZS2VPMmU1TTFl?=
 =?utf-8?B?RTgyUWtUbkp2Nk04bW55cE1YdXBvWGlVaU1KSDJ3b1ZoV01oa0pmaSttVFpT?=
 =?utf-8?B?V3NEZjY3aFpTY25KVFBLY0xJeERVU1VncXBwdkxsQTM0UlNsWFNZUVdnMWNU?=
 =?utf-8?B?d0Fwc25OL05leVpiRlFSbmx5YUVpR2lIdzlKWUJwUXMyMVRoZ2RNUlFGUXFa?=
 =?utf-8?B?Q2IzOFBDVUJndS83cHNhdW9GUEpWblAvY1Q0WTNUZnNPdXBpWTI2RU1ZQmt1?=
 =?utf-8?B?SDMrSjlBeWlyaGtkWVZZYkRGODhvalB0bDI5ajBPRWNnMXQyNWFHaU5mMWY3?=
 =?utf-8?B?T20zNHIwQnNUQWVHMytPa2FoazdraTNTNHZqSnBrbzYwcE5UcUFldlhRdnRQ?=
 =?utf-8?B?RVg3d2NtajRSYVdhREVGSldFdjB0aU9kRzFqUGVJTzRaUXdVblJPVzZZd1hu?=
 =?utf-8?B?K2EvU3ZkZzd5WlNLMkRSNnpZSjZ2VlhUSUdjbEl2NDVTb3piL3dRQ2hGSVBN?=
 =?utf-8?B?MWZrSzE0QitmbU9EaVJnOXVsOS9SQ0JoKzIzcUYzZUNFSWtkTUFyMC92c05l?=
 =?utf-8?B?SHIzSkpLN1lWeWtSREMzazFVa1VJK2F2VHduQ2VjanhpMnc1dVg1WkZRVnNF?=
 =?utf-8?B?Wm0yYk5nUTNWTC9zN0ljOVdGMUoyNWNHbTZjTFhKbVVCc01EcXNYT2lKNXBD?=
 =?utf-8?B?QTVNc3VKcWRzdkk0Y3hXem5FR1hBOVgvSGplNkVJKzdsOWo1TVpyNmFmSjFn?=
 =?utf-8?B?bkNwNWtsRXczbVI0WDlheE40ZHRQL1RzV09pU1N0eUtNNHExcmFFV1doRzAy?=
 =?utf-8?B?OFE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 9f6e0c92-b7e9-4561-204e-08dacc0ba3a5
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Nov 2022 21:59:18.9810
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FwV7vl96GcG53wPpLUF+Y+HWQmR5jyVagmH46uJH9BJ5+hskgtOFTzU5Kk1TKbL5aT5/gSrbMjgCaJaTk88b31QyzQ/4/6oXeQ/OSuzmb/A=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR11MB6232
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Peter,

Patch description in subject should start with upper case.

On 11/15/2022 6:19 AM, Peter Newman wrote:
> When determining whether running tasks need to be interrupted due to a
> closid/rmid change, it was possible for the task in question to migrate
> or wake up concurrently without observing the updated values.

Mixing tenses can quickly become confusing. Please stick to imperative tone. 

Also, please start with the context of this work before jumping to
the problem description.

For example (not a requirement to use - feel free to change):
"A task is moved to a resource group when its task id is written to the
destination resource group's "tasks" file. Moving a task to a new
resource group involves updating the task's closid and rmid (found
in its task_struct) and updating the PQR_ASSOC MSR if the task
is current on a CPU.

It is possible for the task to migrate or wake up while it is moved
to a new resource group. In this scenario the task starts running
with the old closid and rmid values but it may not be considered
as running and thus continue running with the old values until it is
rescheduled."

> 
> In particular, __rdtgroup_move_task() assumed that a CPU migrating
> implied that it observed the updated closid/rmid. This assumption is
> broken by the following reorderings, both of which are allowed on x86:
> 
>  1. In __rdtgroup_move_task(), stores updating the closid and rmid in
>     the task structure could reorder with the loads in task_curr() and
>     task_cpu().
>  2. In resctrl_sched_in(), the earlier stores to the fields read by
>     task_curr() could be delayed until after the loads from
>     t->{closid,rmid}.
> 
> Preventing this reordering with barriers would have required an smp_mb()
> in all context switches whenever resctrlfs is mounted.  Instead, when
> moving a single task, use task_call_func() to serialize updates to the
> closid and rmid fields in the task_struct with context switch.

Please adjust the above to imperative tone.

> 
> Signed-off-by: Peter Newman <peternewman@google.com>
> Reviewed-by: James Morse <james.morse@arm.com>
> ---
>  arch/x86/kernel/cpu/resctrl/rdtgroup.c | 76 +++++++++++++++-----------
>  1 file changed, 45 insertions(+), 31 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> index 049971efea2f..511b7cea143f 100644
> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> @@ -538,10 +538,47 @@ static void _update_task_closid_rmid(void *task)
>  		resctrl_sched_in();
>  }
>  
> -static void update_task_closid_rmid(struct task_struct *t)
> +static int update_locked_task_closid_rmid(struct task_struct *t, void *arg)
>  {
> -	if (IS_ENABLED(CONFIG_SMP) && task_curr(t))
> -		smp_call_function_single(task_cpu(t), _update_task_closid_rmid, t, 1);
> +	struct rdtgroup *rdtgrp = arg;
> +
> +	/*
> +	 * We assume task_call_func() has provided the necessary serialization
> +	 * with resctrl_sched_in().

Please no "we".

Also, either task_call_func() provides serialization or it does not. Wording
it as "assume" creates uncertainty about this change.

> +	 */
> +	if (rdtgrp->type == RDTCTRL_GROUP) {
> +		t->closid = rdtgrp->closid;
> +		t->rmid = rdtgrp->mon.rmid;
> +	} else if (rdtgrp->type == RDTMON_GROUP) {
> +		t->rmid = rdtgrp->mon.rmid;
> +	}

wrt the READ_ONCE() in __resctrl_sched_in() ... memory_barriers.txt tells me
that "When dealing with CPU-CPU interactions, certain types of memory barrier
should always be paired.  A lack of appropriate pairing is almost certainly
an error." 

> +
> +	/*
> +	 * If the task is current on a CPU, the PQR_ASSOC MSR needs to be
> +	 * updated to make the resource group go into effect. If the task is not
> +	 * current, the MSR will be updated when the task is scheduled in.
> +	 */
> +	return task_curr(t);
> +}
> +
> +static void update_task_closid_rmid(struct task_struct *t,
> +				    struct rdtgroup *rdtgrp)
> +{
> +	/*
> +	 * Serialize the closid and rmid update with context switch. If this
> +	 * function indicates that the task was running, then it needs to be

What does "this function" refer to? Please replace with function name to be
specific since there are a few functions below.

/was running/is running/?

> +	 * interrupted to install the new closid and rmid.
> +	 */
> +	if (task_call_func(t, update_locked_task_closid_rmid, rdtgrp) &&
> +	    IS_ENABLED(CONFIG_SMP))
> +		/*
> +		 * If the task has migrated away from the CPU indicated by
> +		 * task_cpu() below, then it has already switched in on the
> +		 * new CPU using the updated closid and rmid and the call below
> +		 * unnecessary, but harmless.

is unnecessary ?

> +		 */
> +		smp_call_function_single(task_cpu(t),
> +					 _update_task_closid_rmid, t, 1);
>  	else
>  		_update_task_closid_rmid(t);
>  }

Could you please keep update_task_closid_rmid() and
_update_task_closid_rmid() together?

> @@ -557,39 +594,16 @@ static int __rdtgroup_move_task(struct task_struct *tsk,
>  		return 0;
>  
>  	/*
> -	 * Set the task's closid/rmid before the PQR_ASSOC MSR can be
> -	 * updated by them.
> -	 *
> -	 * For ctrl_mon groups, move both closid and rmid.
>  	 * For monitor groups, can move the tasks only from
>  	 * their parent CTRL group.
>  	 */
> -
> -	if (rdtgrp->type == RDTCTRL_GROUP) {
> -		WRITE_ONCE(tsk->closid, rdtgrp->closid);
> -		WRITE_ONCE(tsk->rmid, rdtgrp->mon.rmid);
> -	} else if (rdtgrp->type == RDTMON_GROUP) {
> -		if (rdtgrp->mon.parent->closid == tsk->closid) {
> -			WRITE_ONCE(tsk->rmid, rdtgrp->mon.rmid);
> -		} else {
> -			rdt_last_cmd_puts("Can't move task to different control group\n");
> -			return -EINVAL;
> -		}
> +	if (rdtgrp->type == RDTMON_GROUP &&
> +	    rdtgrp->mon.parent->closid != tsk->closid) {
> +		rdt_last_cmd_puts("Can't move task to different control group\n");
> +		return -EINVAL;
>  	}
>  
> -	/*
> -	 * Ensure the task's closid and rmid are written before determining if
> -	 * the task is current that will decide if it will be interrupted.
> -	 */
> -	barrier();
> -
> -	/*
> -	 * By now, the task's closid and rmid are set. If the task is current
> -	 * on a CPU, the PQR_ASSOC MSR needs to be updated to make the resource
> -	 * group go into effect. If the task is not current, the MSR will be
> -	 * updated when the task is scheduled in.
> -	 */
> -	update_task_closid_rmid(tsk);
> +	update_task_closid_rmid(tsk, rdtgrp);
>  
>  	return 0;
>  }

Reinette