Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <54e50a9b-268f-2020-f54c-d38312489e2f@intel.com>
Date:   Mon, 21 Nov 2022 13:53:23 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.5.0
Subject: Re: [PATCH v3 1/2] x86/resctrl: IPI all CPUs for group updates
To:     Peter Newman <peternewman@google.com>, <fenghua.yu@intel.com>
CC:     <bp@alien8.de>, <derkling@google.com>, <eranian@google.com>,
        <hpa@zytor.com>, <james.morse@arm.com>, <jannh@google.com>,
        <kpsingh@google.com>, <linux-kernel@vger.kernel.org>,
        <mingo@redhat.com>, <tglx@linutronix.de>, <x86@kernel.org>
References: <20221115141953.816851-1-peternewman@google.com>
 <20221115141953.816851-2-peternewman@google.com>
Content-Language: en-US
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20221115141953.816851-2-peternewman@google.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Qjh4UGRJZ0t0RXJmM21JQzJTQWF0dGhVa2pUclhzN3FDdDNrL1NyZnpvWSt1?=
 =?utf-8?B?YVpQQ1Jxa2lEMTJjNGRpbVBsRDhNNlhPaU1YS25hT21aVFZWTG9WVEtmakds?=
 =?utf-8?B?ckc5R3lwaitoZ2N4dkxzZ3F2T1hwcFNwNE9aWFJ5SDJsWEE1eU1WT09oUG5H?=
 =?utf-8?B?VmFxUzJSNlV3VDA0TEhKamI5Q1NNUkhWMmVhWmowL0tpLytsNVdRaU9UbWpk?=
 =?utf-8?B?Q2NMN2E4K05Ib0NuVUd3SGE5NU80VnVSd1F6ZkZoaXV3UDh2YkRHVDZKbjJW?=
 =?utf-8?B?eGV2Ri8raDdteG5JSzNuOUNCM3NNaG9NMitWa0swQU11N1pYcWF2bUo1YVQz?=
 =?utf-8?B?WlE3UTNFYlJGZEIxNHJHNGVDSHFRemU5NkZoYnpxM001UHh0WWwxM0QrWkZu?=
 =?utf-8?B?SWpSNHpKRDB3N1JJcTZ1bW81WlRZc1NmMU1QMDBoaSt5eHJ0VkVMaUhMZllu?=
 =?utf-8?B?YndBYUV1dW84VUw0OUhST1dHWDF0ZFcxNTlKL0tyRHVhaFVpZVZaQy9kcVhD?=
 =?utf-8?B?MmEyVXpyeVNCVisvQVdaVjV5bFhTeXBzaU9DS1E0elRoalNCRFJSRkFlQXpZ?=
 =?utf-8?B?VTV4bGRhU0l4N0JxZ2FsYm04VUd6NkZwY2Q4Y2RrT1RhYXZNQTR4ang0dGdI?=
 =?utf-8?B?TDRCSVpQQU83M0FqckdBN2tuREJBaG5sL3FGZ2RLTlBTTDRMZmdzN2Q1d01L?=
 =?utf-8?B?VTVHdDduK0R2ZHBNYXB1OWxJZ0paZ0w4Q2xKeVpoVTJKNlBBVzFQMXJsRnZH?=
 =?utf-8?B?NXpOV1Fxb2txZmhoVGU0UDlPM0RJUkkvekJydHIvOVZtZC9EU3hhNmI3WWFB?=
 =?utf-8?B?WElHZkROeU0zODJhcGhjRVRDQzZvMlpzQVFaQVBxb1RDSDF6Mi9nbFVXZW1H?=
 =?utf-8?B?c1J5bmNGZDBJeDhCM2srclRqTTBzSlZRc0dvTkFGVkdOVW0veGhsNS9ZdUx6?=
 =?utf-8?B?UmNrbEdQQ0JmbVk1dHZHM2diSS90TmIrYmh6WXJJczdPbXZPaWhRaTV2RjVr?=
 =?utf-8?B?UlFucFFlNUV4dTdnZnBBOUtTK01LR1QyYXNJaFgyRjVlSFpYUFMxSkVmNndw?=
 =?utf-8?B?TVZlUTgrc0U4UGhCcnduN2VnU2N3VGIzQWVzZWN4bi9qT2VVd3NCQmFVYi9J?=
 =?utf-8?B?SHByaHcxR0w0Sk96NUxhTVZESHFKU2c0eEVpbzk3cmNObDZndXR3cWVWOG1h?=
 =?utf-8?B?N0c4Vkh0U3B3b053bkJhanduaHhObWhKQU5sbHhDSmo2Q2NiM2tLT1l0Um11?=
 =?utf-8?B?Q2ZhSUhodjVRS1Y0WHI3RjVyV29tUzFIWVJmMjh0czJrZ2pDNzgvYVBFVVY5?=
 =?utf-8?B?NnlRL0s2VlFZSEF6bW9iekc1RWM1bzRXeEFSTTc5anR0REhoZCtTbHFmRXk4?=
 =?utf-8?B?bUMxenhJMFNNVUc0bUdiZ1gvWFBNS1pobUdVZnhZbmVlZVBJWW12WWgxemw2?=
 =?utf-8?B?cTV2S3RqOElqazFTbXZxdENmSWNtV1p3NTVMcm5RVWk4SDIvM2k5ckZOQjZQ?=
 =?utf-8?B?Y0xZeGtFMGgwem4vUzlBcHZtNjFZbDFPTmN6Kys1TzgvZVUxRDNwVGxPUGZk?=
 =?utf-8?B?M2hodThHaDVJRWtsUk1FWFpac2F3ZnJYMkZGZkltR2N3S1ZBR0dSTHlKS05S?=
 =?utf-8?B?TlFhWkJHUXBNSGNaVDBoS0owZ2FFT3pUTzdSZnBLaG9sRzhTT0txZlhIeFBv?=
 =?utf-8?B?UkZaNlpDU2ltbGRqbk1Bd1pzQzlRRGtxZ1IzVVVQS2VCdGtQNGpLNnU1bmFU?=
 =?utf-8?B?alpZczFjWVV4Snk3c0xiNDMxdkkveElIL0VvYWdhbWNHQjh3YkVYV25oZ2VF?=
 =?utf-8?B?VHBXdmg4bzJlSVF4NDhoYUF5eWJPclFKTnhRTFRSRE9uTGZIQlREcExBMzll?=
 =?utf-8?B?bWM0ZFkxU1VCOHlaZFkxV3NTVXM4QXAvRit2dGl4ZFd6K0RkRFBqa0hhRDI1?=
 =?utf-8?B?dUdSTlZiNlBIUzY1SFFxQ0RPSjNpbjVsK29ubVkvOE56MkxzNkc5ZjR3TWJj?=
 =?utf-8?B?VjljVDg1dThuN1NoeUtoL1NiL0U3bFFkUE1hU0F3YStpNWJrQ09mc3Y4Zm9y?=
 =?utf-8?B?ZW5Ia296eURUV3RFc1JPeExrT3hCNEZHaDcxVHlUUnJBMUhJblJRL0FSbzUv?=
 =?utf-8?B?eGV0bFlCK25IRFZOUWt4SEdjWEYvMzAzWk1TYjY1WWhwYmJRenoxQUVPME9B?=
 =?utf-8?B?bVE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b63e718a-569f-4023-993d-08dacc0ad1ad
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Nov 2022 21:53:26.7931
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3CO4TsGCULbsxDwR5zVzbw0gu9iHPZwqyrEmijIQLjI5WwcXuUPHGWOWzyhbm2tQ7xHkIlB6UdhKOXUiYqEM04zruMpV6u0xgth+m8WKnOU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB4846
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Peter,

On 11/15/2022 6:19 AM, Peter Newman wrote:
> To rule out needing to update a CPU when deleting an rdtgroup, we must

Please do not impersonate code in changelog and comments (do not
use "we"). This is required for resctrl changes to be considered for
inclusion because resctrl patches are routed via the "tip" repo
and is thus required to follow the "tip tree handbook" 
(Documentation/process/maintainer-tip.rst). Please also
stick to a clear "context-problem-solution" changelog as is the custom
in this area.

> search the entire tasklist for group members which could be running on
> that CPU. This needs to be done while blocking updates to the tasklist
> to avoid leaving newly-created child tasks assigned to the old
> CLOSID/RMID.

This is not clear to me. rdt_move_group_tasks() obtains a read lock,
read_lock(&tasklist_lock), so concurrent modifications to the tasklist
are indeed possible. Should this perhaps be write_lock() instead?
It sounds like the scenario you are describing may be a concern. That is,
if a task belonging to a group that is being removed happens to
call fork()/clone() during the move then the child may end up being
created with old closid.

> 
> The cost of reliably propagating a CLOSID or RMID update to a single
> task is higher than originally thought. The present understanding is
> that we must obtain the task_rq_lock() on each task to ensure that it
> observes CLOSID/RMID updates in the case that it migrates away from its
> current CPU before the update IPI reaches it.

I find this confusing since it describes why a potential solution does
not solve a problem, neither problem nor solution is well described at this
point. 
 
What if you switch the order of the two patches? Patch #2 provides
the potential solution mentioned here so that may be helpful to have as
reference in this changelog.

> For now, just notify all the CPUs after updating the closid/rmid fields

For now? If you anticipate changes then there should be a plan for that, 
otherwise this is the fix without further speculation.

> in impacted tasks task_structs rather than paying the cost of obtaining
> a more precise cpu mask.

s/cpu/CPU/
It may be helpful to add that an accurate CPU mask cannot be guaranteed and
the more tasks moved the less accurate it could be (if I understand correctly).

> 
> Signed-off-by: Peter Newman <peternewman@google.com>
> Reviewed-by: James Morse <james.morse@arm.com>
> ---
>  arch/x86/kernel/cpu/resctrl/rdtgroup.c | 52 +++++++-------------------
>  1 file changed, 13 insertions(+), 39 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> index e5a48f05e787..049971efea2f 100644
> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> @@ -2385,12 +2385,10 @@ static int reset_all_ctrls(struct rdt_resource *r)
>   * Move tasks from one to the other group. If @from is NULL, then all tasks
>   * in the systems are moved unconditionally (used for teardown).
>   *
> - * If @mask is not NULL the cpus on which moved tasks are running are set
> - * in that mask so the update smp function call is restricted to affected
> - * cpus.
> + * Following this operation, the caller is required to update the MSRs on all
> + * CPUs.
>   */

On x86 only one MSR needs updating, the PQR_ASSOC MSR. The above could be
summarized as:
"Caller should update per CPU storage and PQR_ASSOC."

> -static void rdt_move_group_tasks(struct rdtgroup *from, struct rdtgroup *to,
> -				 struct cpumask *mask)
> +static void rdt_move_group_tasks(struct rdtgroup *from, struct rdtgroup *to)
>  {
>  	struct task_struct *p, *t;
>  
> @@ -2400,16 +2398,6 @@ static void rdt_move_group_tasks(struct rdtgroup *from, struct rdtgroup *to,
>  		    is_rmid_match(t, from)) {
>  			WRITE_ONCE(t->closid, to->closid);
>  			WRITE_ONCE(t->rmid, to->mon.rmid);
> -
> -			/*
> -			 * If the task is on a CPU, set the CPU in the mask.
> -			 * The detection is inaccurate as tasks might move or
> -			 * schedule before the smp function call takes place.
> -			 * In such a case the function call is pointless, but
> -			 * there is no other side effect.
> -			 */
> -			if (IS_ENABLED(CONFIG_SMP) && mask && task_curr(t))
> -				cpumask_set_cpu(task_cpu(t), mask);
>  		}
>  	}
>  	read_unlock(&tasklist_lock);
> @@ -2440,7 +2428,7 @@ static void rmdir_all_sub(void)
>  	struct rdtgroup *rdtgrp, *tmp;
>  
>  	/* Move all tasks to the default resource group */
> -	rdt_move_group_tasks(NULL, &rdtgroup_default, NULL);
> +	rdt_move_group_tasks(NULL, &rdtgroup_default);
>  
>  	list_for_each_entry_safe(rdtgrp, tmp, &rdt_all_groups, rdtgroup_list) {
>  		/* Free any child rmids */
> @@ -3099,23 +3087,19 @@ static int rdtgroup_mkdir(struct kernfs_node *parent_kn, const char *name,
>  	return -EPERM;
>  }
>  
> -static int rdtgroup_rmdir_mon(struct rdtgroup *rdtgrp, cpumask_var_t tmpmask)
> +static int rdtgroup_rmdir_mon(struct rdtgroup *rdtgrp)
>  {
>  	struct rdtgroup *prdtgrp = rdtgrp->mon.parent;
>  	int cpu;
>  
>  	/* Give any tasks back to the parent group */
> -	rdt_move_group_tasks(rdtgrp, prdtgrp, tmpmask);
> +	rdt_move_group_tasks(rdtgrp, prdtgrp);
>  
>  	/* Update per cpu rmid of the moved CPUs first */
>  	for_each_cpu(cpu, &rdtgrp->cpu_mask)
>  		per_cpu(pqr_state.default_rmid, cpu) = prdtgrp->mon.rmid;
> -	/*
> -	 * Update the MSR on moved CPUs and CPUs which have moved
> -	 * task running on them.
> -	 */
> -	cpumask_or(tmpmask, tmpmask, &rdtgrp->cpu_mask);
> -	update_closid_rmid(tmpmask, NULL);
> +
> +	update_closid_rmid(cpu_online_mask, NULL);
>  
>  	rdtgrp->flags = RDT_DELETED;
>  	free_rmid(rdtgrp->mon.rmid);
> @@ -3140,12 +3124,12 @@ static int rdtgroup_ctrl_remove(struct rdtgroup *rdtgrp)
>  	return 0;
>  }
>  
> -static int rdtgroup_rmdir_ctrl(struct rdtgroup *rdtgrp, cpumask_var_t tmpmask)
> +static int rdtgroup_rmdir_ctrl(struct rdtgroup *rdtgrp)
>  {
>  	int cpu;
>  
>  	/* Give any tasks back to the default group */
> -	rdt_move_group_tasks(rdtgrp, &rdtgroup_default, tmpmask);
> +	rdt_move_group_tasks(rdtgrp, &rdtgroup_default);
>  
>  	/* Give any CPUs back to the default group */
>  	cpumask_or(&rdtgroup_default.cpu_mask,
> @@ -3157,12 +3141,7 @@ static int rdtgroup_rmdir_ctrl(struct rdtgroup *rdtgrp, cpumask_var_t tmpmask)
>  		per_cpu(pqr_state.default_rmid, cpu) = rdtgroup_default.mon.rmid;
>  	}
>  
> -	/*
> -	 * Update the MSR on moved CPUs and CPUs which have moved
> -	 * task running on them.
> -	 */
> -	cpumask_or(tmpmask, tmpmask, &rdtgrp->cpu_mask);
> -	update_closid_rmid(tmpmask, NULL);
> +	update_closid_rmid(cpu_online_mask, NULL);
>  
>  	closid_free(rdtgrp->closid);
>  	free_rmid(rdtgrp->mon.rmid);
> @@ -3181,12 +3160,8 @@ static int rdtgroup_rmdir(struct kernfs_node *kn)
>  {
>  	struct kernfs_node *parent_kn = kn->parent;
>  	struct rdtgroup *rdtgrp;
> -	cpumask_var_t tmpmask;
>  	int ret = 0;
>  
> -	if (!zalloc_cpumask_var(&tmpmask, GFP_KERNEL))
> -		return -ENOMEM;
> -
>  	rdtgrp = rdtgroup_kn_lock_live(kn);
>  	if (!rdtgrp) {
>  		ret = -EPERM;
> @@ -3206,18 +3181,17 @@ static int rdtgroup_rmdir(struct kernfs_node *kn)
>  		    rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED) {
>  			ret = rdtgroup_ctrl_remove(rdtgrp);
>  		} else {
> -			ret = rdtgroup_rmdir_ctrl(rdtgrp, tmpmask);
> +			ret = rdtgroup_rmdir_ctrl(rdtgrp);
>  		}
>  	} else if (rdtgrp->type == RDTMON_GROUP &&
>  		 is_mon_groups(parent_kn, kn->name)) {
> -		ret = rdtgroup_rmdir_mon(rdtgrp, tmpmask);
> +		ret = rdtgroup_rmdir_mon(rdtgrp);
>  	} else {
>  		ret = -EPERM;
>  	}
>  
>  out:
>  	rdtgroup_kn_unlock(kn);
> -	free_cpumask_var(tmpmask);
>  	return ret;
>  }
>  

The fix looks good to me. I do think its motivation and description
needs to improve to make it palatable to folks not familiar with this area.

Reinette