Received-SPF: pass (google.com: domain of linux-kernel+bounces-15924-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <286216a9-d586-42dc-9f1a-8909364bd7bd@amd.com>
Date: Wed, 3 Jan 2024 13:43:01 -0600
User-Agent: Mozilla Thunderbird
From: "Moger, Babu" <babu.moger@amd.com>
Subject: Re: [PATCH v8 08/24] x86/resctrl: Track the number of dirty RMID a
 CLOSID has
Reply-To: babu.moger@amd.com
To: James Morse <james.morse@arm.com>, x86@kernel.org,
 linux-kernel@vger.kernel.org
Cc: Fenghua Yu <fenghua.yu@intel.com>,
 Reinette Chatre <reinette.chatre@intel.com>,
 Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>,
 Borislav Petkov <bp@alien8.de>, H Peter Anvin <hpa@zytor.com>,
 shameerali.kolothum.thodi@huawei.com,
 D Scott Phillips OS <scott@os.amperecomputing.com>,
 carl@os.amperecomputing.com, lcherian@marvell.com,
 bobo.shaobowang@huawei.com, tan.shaopeng@fujitsu.com,
 baolin.wang@linux.alibaba.com, Jamie Iles <quic_jiles@quicinc.com>,
 Xin Hao <xhao@linux.alibaba.com>, peternewman@google.com,
 dfustini@baylibre.com, amitsinght@marvell.com
References: <20231215174343.13872-1-james.morse@arm.com>
 <20231215174343.13872-9-james.morse@arm.com>
Content-Language: en-US
In-Reply-To: <20231215174343.13872-9-james.morse@arm.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?VnJhMElndlVsd2VWem5iUCtBNHY1UGw0bXQvcVFUVXNKQ3dIeithRllTZy81?=
 =?utf-8?B?eTlwR3pZYUU5Si9Yc2NwK2pPWlUweFJiR29EM1NZUnY0SzdJODd4MTFYeklh?=
 =?utf-8?B?cGtZOTg3RkdCVTlLN2hWVUNvcFBrUUY2eXBRUHB0aUZqaFRHdUN4VW4xWkUz?=
 =?utf-8?B?NW9DczRTRFFuOFIzeWwxNEc2RTBUcTR2Wld4MDZzWUN5RmZWV2RJYXBQR2M2?=
 =?utf-8?B?aWcydTJSUU9Wak1WaXdNa1lkL0c2SGRWWHJMcVV1WTZLcGpORVZtQm5GOVY4?=
 =?utf-8?B?TzN6WjArekw5cyszenA1TEJ3bDVOUFhBOFNBdEVLRlZmeEZJTTJmME43YjVt?=
 =?utf-8?B?NHNQa21OU0wzM1ZQV0t0cXloaVpQNk15Qm44dEhhd2trRUtZYnZmK3ZMZDI0?=
 =?utf-8?B?QWhJRWdGNnBZVWlISjJTc1Y4MjJQQ0dPcVp2d1V5TUZBZ0lKdHZWd3dEUkRE?=
 =?utf-8?B?Wk5EYzBLUUJ3bnVlcFFGdHV2cU0vZDNnQk8wejg3Q01TVUN5SEJvZklubGhw?=
 =?utf-8?B?RmtxRkRYY3dyemppQitxcmdOODhuakViOXJUdnBRVG95ckJBc2pyUkRqaEo1?=
 =?utf-8?B?N3YzMm5LSkpFaVllRnRXRy9uSFpOMkx2dEYzYlpVQlhwSWtCcFRESWphclpn?=
 =?utf-8?B?cXhPVERkYWpqZWhhRyszZVhCaUhFNnhnelhuaW9tSWhtTXlnYWZTL3hEbzRT?=
 =?utf-8?B?VEtab29SZ3VCS0lHdjB1Q1lab2libnB1aVg1Nm9xTUpiRjVmeDJiVzgwVmJH?=
 =?utf-8?B?b2pDSUpKY3Eyb2xoUCtDOWtvQ1FibDM3VjVSMGw1RnY3UjhwNDRSYVo1ejFa?=
 =?utf-8?B?SDZHRGZXOFFNak12aFhmb0FZNjhFU3RuTjNxZVA4ZTZNYnlzcU1UL01uU3hH?=
 =?utf-8?B?OXQzTUU2ZWxWQ0ZNZk4vMGlaTDYyUEN2WSs1b3VUa2JZU242eEg5ZTRqR3h0?=
 =?utf-8?B?KzBST0hKVVdob0sva0k2d205ZVZ1bjc3d1pTaURaTG85bThpREJzUFhkRmtH?=
 =?utf-8?B?S04zd3B1bGJBVHBXTnFmMTlLeEJIcGVqTDQ4TnYxYkVJZFpYNFo3VlpnUklx?=
 =?utf-8?B?Ny9JWnlHbmg2VjhUY1VVemtkRzdKcG1lQzNQdWplN1BUL09HWFJUQnJoNnpD?=
 =?utf-8?B?d0l0b1RGeFAvZkZYbGZmOTdSNS85NWJacHlUSE9YSjU4YklMNzg0Wk1teFhL?=
 =?utf-8?B?VitTZ1QwSFV3cFpucWI1dUkxQkZ1SlVRRkNUMjVuMmxPQnEyQTdMSlovQmxC?=
 =?utf-8?B?cmdUNVBSNU1TQ2RvRk1lNmQ4b3N6a1hVUXczamJrRkovb3cvbXV4ZEZXZGF2?=
 =?utf-8?B?NFZwZDgvOFFycVEvYkFFSUI4eFhuSzAyMWg3dFhSRTlwTk1PL2tjeDk2NkVj?=
 =?utf-8?B?RzRRaUk2Rmd4eGJGanpQeEc3aVZjUElVQ1FuNUdmSTQyOS84cVNWOE8yZXVh?=
 =?utf-8?B?endTOTVySFF3akw0M3dQa3pjandxOTg1QnpjRVZtT2VycDVpUmFHMHRZRi9R?=
 =?utf-8?B?Ynd3aDNtMXk0TlRsQ3FONGN1RlNkNFNjay9OeW5qQXhzMUI2Vk1pTDRXSmhj?=
 =?utf-8?B?NlI4MDlPS3FVYnlObmtGc05wN1AwSERsQ1BuVzU5cWR5V3JzY3g5M2xUR0ZB?=
 =?utf-8?B?dENyREM0SjYrcnplQThESm4wN1A5UkxPdWhkVE0yRW9CbjNDVVJrT0JZRUFU?=
 =?utf-8?B?TFd2OG50eU4vR1I1bmd0alFsQ3d1blZGYW11cnRIeUtJWTNWK3NTNUY5SEVx?=
 =?utf-8?B?d2JMSVBtVXFaeFB2OCtaZkY5MTFhRThyL1VFNzJBYTVYM2JOMC93V3I5RXA2?=
 =?utf-8?B?NmJQYUwvb0tGRGdUZHlnT3BHd0k3Z1hrd2M3SjRxUm5IYklQYUhsSnBSRXlk?=
 =?utf-8?B?YmxrSFVkWUpFTFV6aHg5S0xlVTllcStCcmxUcTZCQXFPZ1YwczVxazlPTVdE?=
 =?utf-8?B?d2IvMStIZm95VmFXa2IrbTN6cEs4YW5qWUdmajFXZWg5NnBDSWJyWDZ0UnRD?=
 =?utf-8?B?RFczYnYwTFB3ZTk2czB2ZTJUNlJoQ2dUME91L25uM2kyTmNtUGlXeGZhRDZR?=
 =?utf-8?B?ajhXMWVtWTc1VjVpUGkrUDhQbXNueGd3MGY4bS9Xc3JaUC85TE00T3ZOeG8y?=
 =?utf-8?Q?MkqE=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6cdef16f-007d-42e0-c1e7-08dc0c94335f
X-MS-Exchange-CrossTenant-AuthSource: MW3PR12MB4553.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2024 19:43:03.7549
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 5bWy/R4Dk7cyQstzXrouMJMaMEz1tdANmrILKfevpWNa5EL4hTuAKSK2yCehrUGi
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB5813


On 12/15/23 11:43, James Morse wrote:
> MPAM's PMG bits extend its PARTID space, meaning the same PMG value can be
> used for different control groups.
> 
> This means once a CLOSID is allocated, all its monitoring ids may still be
> dirty, and held in limbo.
> 
> Keep track of the number of RMID held in limbo each CLOSID has. This will
> allow a future helper to find the 'cleanest' CLOSID when allocating.
> 
> The array is only needed when CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID is
> defined. This will never be the case on x86.
> 
> Signed-off-by: James Morse <james.morse@arm.com>
> Tested-by: Shaopeng Tan <tan.shaopeng@fujitsu.com>
> Tested-by: Peter Newman <peternewman@google.com>
> Tested-by: Babu Moger <babu.moger@amd.com>
> Reviewed-by: Shaopeng Tan <tan.shaopeng@fujitsu.com>
> Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>

Reviewed-by: Babu Moger <babu.moger@amd.com>

> ---
> Changes since v4:
>  * Moved closid_num_dirty_rmid[] update under entry->busy check
>  * Take the mutex in dom_data_init() as the caller doesn't.
> 
> Changes since v5:
>  * Added braces after an else.
>  * Made closid_num_dirty_rmid an unsigned int.
>  * Moved mutex_lock() in dom_data_init() to cover the whole function.
> 
> Changes since v6:
>  * Made closid_num_dirty_rmid[] and associated tmp variables u32.
> 
> Changes since v7:
>  * Clobber kfree()d variable with NULL.
>  * Guard the use of closid_num_dirty_rmid with IS_ENABLED() so it can be
>    optimised out on x86.
> ---
>  arch/x86/kernel/cpu/resctrl/monitor.c | 69 +++++++++++++++++++++++----
>  1 file changed, 59 insertions(+), 10 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
> index 1f371b108a74..6dfc68c800c8 100644
> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
> @@ -50,6 +50,13 @@ struct rmid_entry {
>   */
>  static LIST_HEAD(rmid_free_lru);
>  
> +/**
> + * @closid_num_dirty_rmid    The number of dirty RMID each CLOSID has.
> + *     Only allocated when CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID is defined.
> + *     Indexed by CLOSID. Protected by rdtgroup_mutex.
> + */
> +static u32 *closid_num_dirty_rmid;
> +
>  /*
>   * @rmid_limbo_count - count of currently unused but (potentially)
>   *     dirty RMIDs.
> @@ -292,6 +299,17 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain *d,
>  	return 0;
>  }
>  
> +static void limbo_release_entry(struct rmid_entry *entry)
> +{
> +	lockdep_assert_held(&rdtgroup_mutex);
> +
> +	rmid_limbo_count--;
> +	list_add_tail(&entry->list, &rmid_free_lru);
> +
> +	if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID))
> +		closid_num_dirty_rmid[entry->closid]--;
> +}
> +
>  /*
>   * Check the RMIDs that are marked as busy for this domain. If the
>   * reported LLC occupancy is below the threshold clear the busy bit and
> @@ -328,10 +346,8 @@ void __check_limbo(struct rdt_domain *d, bool force_free)
>  
>  		if (force_free || !rmid_dirty) {
>  			clear_bit(idx, d->rmid_busy_llc);
> -			if (!--entry->busy) {
> -				rmid_limbo_count--;
> -				list_add_tail(&entry->list, &rmid_free_lru);
> -			}
> +			if (!--entry->busy)
> +				limbo_release_entry(entry);
>  		}
>  		cur_idx = idx + 1;
>  	}
> @@ -398,6 +414,8 @@ static void add_rmid_to_limbo(struct rmid_entry *entry)
>  	u64 val = 0;
>  	u32 idx;
>  
> +	lockdep_assert_held(&rdtgroup_mutex);
> +
>  	idx = resctrl_arch_rmid_idx_encode(entry->closid, entry->rmid);
>  
>  	entry->busy = 0;
> @@ -423,10 +441,13 @@ static void add_rmid_to_limbo(struct rmid_entry *entry)
>  	}
>  	put_cpu();
>  
> -	if (entry->busy)
> +	if (entry->busy) {
>  		rmid_limbo_count++;
> -	else
> +		if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID))
> +			closid_num_dirty_rmid[entry->closid]++;
> +	} else {
>  		list_add_tail(&entry->list, &rmid_free_lru);
> +	}
>  }
>  
>  void free_rmid(u32 closid, u32 rmid)
> @@ -792,13 +813,33 @@ void mbm_setup_overflow_handler(struct rdt_domain *dom, unsigned long delay_ms)
>  static int dom_data_init(struct rdt_resource *r)
>  {
>  	u32 idx_limit = resctrl_arch_system_num_rmid_idx();
> +	u32 num_closid = resctrl_arch_get_num_closid(r);
>  	struct rmid_entry *entry = NULL;
> +	int err = 0, i;
>  	u32 idx;
> -	int i;
> +
> +	mutex_lock(&rdtgroup_mutex);
> +	if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
> +		u32 *tmp;
> +
> +		tmp = kcalloc(num_closid, sizeof(*tmp), GFP_KERNEL);
> +		if (!tmp) {
> +			err = -ENOMEM;
> +			goto out_unlock;
> +		}
> +
> +		closid_num_dirty_rmid = tmp;
> +	}
>  
>  	rmid_ptrs = kcalloc(idx_limit, sizeof(struct rmid_entry), GFP_KERNEL);
> -	if (!rmid_ptrs)
> -		return -ENOMEM;
> +	if (!rmid_ptrs) {
> +		if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
> +			kfree(closid_num_dirty_rmid);
> +			closid_num_dirty_rmid = NULL;
> +		}
> +		err = -ENOMEM;
> +		goto out_unlock;
> +	}
>  
>  	for (i = 0; i < idx_limit; i++) {
>  		entry = &rmid_ptrs[i];
> @@ -818,13 +859,21 @@ static int dom_data_init(struct rdt_resource *r)
>  	entry = __rmid_entry(idx);
>  	list_del(&entry->list);
>  
> -	return 0;
> +out_unlock:
> +	mutex_unlock(&rdtgroup_mutex);
> +
> +	return err;
>  }
>  
>  static void __exit dom_data_exit(struct rdt_resource *r)
>  {
>  	mutex_lock(&rdtgroup_mutex);
>  
> +	if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
> +		kfree(closid_num_dirty_rmid);
> +		closid_num_dirty_rmid = NULL;
> +	}
> +
>  	kfree(rmid_ptrs);
>  	rmid_ptrs = NULL;
>  

-- 
Thanks
Babu Moger