Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <4d8773f7-4219-0bc5-e7df-1be81cba05e7@intel.com>
Date:   Mon, 14 Aug 2023 19:37:35 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.11.0
Subject: Re: [PATCH v5 06/24] x86/resctrl: Track the number of dirty RMID a
 CLOSID has
Content-Language: en-US
To:     James Morse <james.morse@arm.com>, <x86@kernel.org>,
        <linux-kernel@vger.kernel.org>
CC:     Reinette Chatre <reinette.chatre@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        H Peter Anvin <hpa@zytor.com>,
        Babu Moger <Babu.Moger@amd.com>,
        <shameerali.kolothum.thodi@huawei.com>,
        "D Scott Phillips OS" <scott@os.amperecomputing.com>,
        <carl@os.amperecomputing.com>, <lcherian@marvell.com>,
        <bobo.shaobowang@huawei.com>, <tan.shaopeng@fujitsu.com>,
        <xingxin.hx@openanolis.org>, <baolin.wang@linux.alibaba.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Xin Hao <xhao@linux.alibaba.com>, <peternewman@google.com>,
        <dfustini@baylibre.com>
References: <20230728164254.27562-1-james.morse@arm.com>
 <20230728164254.27562-7-james.morse@arm.com>
From:   Fenghua Yu <fenghua.yu@intel.com>
In-Reply-To: <20230728164254.27562-7-james.morse@arm.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cGx2SDBrTSt2YU93Q2t4TEg3bkRONTVraCtwTnhLOGFTYVZqSHhzM3VlSHM4?=
 =?utf-8?B?UExRbHlHa3NwWmVWWXp4bFpFN1JpdkhQYU9FWXF0VTAzRHR5UWRzRld0L09u?=
 =?utf-8?B?eDAyaXNrazRDOG9qalorRmhtRHBzVGE0QWp0SHY2WXpIWU1xb2tMbnNTYVZ2?=
 =?utf-8?B?ekxuNE90a1ZQelUyQUJQS3RTemQrQmdNYlpvNVZWZE5oZUFpa1FqMjNObVpF?=
 =?utf-8?B?TzcrUzF5Z3JoY2dYV0VBSUpsWkUxalFCcUZ3VS9WL1hSYmZhK2FJSVM5aExJ?=
 =?utf-8?B?bkVIai9XWlNQZzRyUDNCWTZMYmRFZXNZUjFYSGxBV3IrTkdxUDhudG1LbmpE?=
 =?utf-8?B?SVoxMURqMGl1TW1BR1FNcmd1L3h5R29YNGJ4S1lrQVlISHV1Z2lHN1ROaWJC?=
 =?utf-8?B?TFM3aXpSc3JnNVJiRVNrM2o1Q1A1SEdQMkEybDdhakFjc05vQTltaGhEbnhH?=
 =?utf-8?B?MTJOQXcxeWFzVEZhOEc5bzZwVWVUSVBVZE42UEt1bVAwU0NsVWhsKzlwV3Fu?=
 =?utf-8?B?VUNaYmNIWmJwdWFvdU94T3JVamUxemNTQ1ZNQUFDUys3emJMUGlqYkV0dUVl?=
 =?utf-8?B?Z2RQd0Yrb1FOd2x0OFFpaGI3c3c4bFpvcEdveTZGUU5LeDI4bi92TUNjZ1hv?=
 =?utf-8?B?c0twbnhiS0FUVXJVSHl1ZWFkbThCMU1yNnV1YVV1cm9NR2F0U056akFHWExu?=
 =?utf-8?B?OUNHSC9JQjZGZ3RmYzcrcVhoTnZZZVgwbitmVloxTUpPMDAyRDdNVldPVzJQ?=
 =?utf-8?B?b0doZml6MTk0RTZyaGlQbUhZY3NvSHJ3UmhhemFXSGdkdzZZclRYc2xXSyti?=
 =?utf-8?B?T0swcTVvTkY0Yi9IWDdvR1Z5YXJETnlrWlhwMjRqMFU5Mi91K0RpTFlPWWRh?=
 =?utf-8?B?bTN3R0dsUXZRbUtDRHNhN1hGVU4zcCtCWFdWMW9xNk1sRE1vblN6czRZOXNS?=
 =?utf-8?B?UjRZMHlmMWY1KzZxdlVnRHZNblp4NUhvSTFjMk85Z0NEbnNuYVVzbC8yayth?=
 =?utf-8?B?S3FmTkdtdncrUHoyLzlpM3NEeGZKZnBoaGhtZS8rcFpzRWhWbXFQOGZ3cXpo?=
 =?utf-8?B?b2UyZVNLekRqU3AwM004aXBDTyt0UkRiWnFEWjYwNTVvSTNZM1dDcHF0ellj?=
 =?utf-8?B?SUlxRHFramxiRlZnVUs0V3dKK0M0S3BLNjcvWS9GMEhBSE5WcUIrUnZGUzkr?=
 =?utf-8?B?ODVLdEtZdC9TaFN6TGw5WlNxWnY5aU9ORjNjbElZYzRGdU9kRTJvR0hBcStv?=
 =?utf-8?B?OFpOVm92VU5EYnRTTGUyaG5IWHExbWJuZjF0WXBYZHdqMUttSEtWbDhtdm8x?=
 =?utf-8?B?STN4RlNuM2E3ZGFjTlRPMVlCWG9HdkZkb1UrT2VQMmZoYmFjdFoxZHdpVEZO?=
 =?utf-8?B?a1MyZDlKV1NRbmd0c0dsRWlzMUozS1NGVUZOdm5sUHVQNnpXTE14QS9yL0l2?=
 =?utf-8?B?L1gzc0swMUtXWU9ETlloWWJhM1p4ZWQvRmhpemdEM2Q3Y1RTcG15NDlmRW1i?=
 =?utf-8?B?bE5ZVWdwNzQxNi8yWHQ5TXB4cUEycjhRZkRhcGVpdzUwWTArYUZEVkdTcG9h?=
 =?utf-8?B?VFNKMGVkNi9iVmwwRU1wQURxRENJMWFXT1dvNXg0UlZjN2ZBbDRJM0JGZ2g2?=
 =?utf-8?B?YTVleDI4UFljTjE3OThjdnE4WDVSTEluOCtDSHF2N3luZjJiYmxOWjd3cU1T?=
 =?utf-8?B?WU9aSXJPT0x6Ukx5ZmMvcU0xMWRkd0IrNGE0alNLMllYKzBVZ2Z0bzc2aFVF?=
 =?utf-8?B?dGs0TVRQVVNzbTRXRzlYazg1dnVIaDVBWUh1MjMvV3IrbVV1UHVEZXpEbEh6?=
 =?utf-8?B?RFN5aEptYm9ISTFycnFQRHh4Ky9FQUFnSVoraGZrS25PTmwzMFhXUGZXWTdJ?=
 =?utf-8?B?Ym9aS2ZLZFJNNU9aTzZmMnJQaEhWNEJ1RTRUanhUWlBFVmNyR3lXR2M0TGRB?=
 =?utf-8?B?bmJ0aGppM1c0dTdnbG1WcExJUEgzNHU3NzV5TWpES2VDSmFuQ2lWVmlKK3dM?=
 =?utf-8?B?bk5NZWYyU2REdzJabnA1V3Yvd0FyV2czQS9IS3JGd0J6SnlLQXVXU3VLSHBB?=
 =?utf-8?B?ZUR6ZXl4ZTZUUkZhREFIZyt6MWE0YUEybWYwRkNWbVlTN0FCeE1nR2dVUUth?=
 =?utf-8?Q?PTHXGb+4670J61wiKCVhJCZUo?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 685f4b2b-3809-4c4b-9b63-08db9d38900b
X-MS-Exchange-CrossTenant-AuthSource: IA1PR11MB6097.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Aug 2023 02:37:26.7154
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: tdOWEdc6DEQpTQNr8JgzYnMFJH8334IsVjCdqeKNB9DAnTGIs8GLIWnVZimZfHnpgD51CIz8TCYpslOmS8rwwQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR11MB6277
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-6.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi, James,

On 7/28/23 09:42, James Morse wrote:
> MPAM's PMG bits extend its PARTID space, meaning the same PMG value can be
> used for different control groups.
> 
> This means once a CLOSID is allocated, all its monitoring ids may still be
> dirty, and held in limbo.
> 
> Keep track of the number of RMID held in limbo each CLOSID has. This will
> allow a future helper to find the 'cleanest' CLOSID when allocating.
> 
> The array is only needed when CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID is
> defined. This will never be the case on x86.
> 
> Signed-off-by: James Morse <james.morse@arm.com>
> ---
> Changes since v4:
>   * Moved closid_num_dirty_rmid[] update under entry->busy check
>   * Take the mutex in dom_data_init() as the caller doesn't.
> ---
>   arch/x86/kernel/cpu/resctrl/monitor.c | 49 +++++++++++++++++++++++----
>   1 file changed, 42 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
> index de91ca781d9f..44addc0126fc 100644
> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
> @@ -43,6 +43,13 @@ struct rmid_entry {
>    */
>   static LIST_HEAD(rmid_free_lru);
>   

Better to add:

#if CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID
> +/**
> + * @closid_num_dirty_rmid    The number of dirty RMID each CLOSID has.
> + * Only allocated when CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID is defined.
> + * Indexed by CLOSID. Protected by rdtgroup_mutex.
> + */
> +static int *closid_num_dirty_rmid;
#endif

Then the global variable won't exist on x86 to avoid confusion and space.

Some code related to the CONFIG also needs to be changed accordingly.

> +
>   /**
>    * @rmid_limbo_count     count of currently unused but (potentially)
>    *     dirty RMIDs.
> @@ -285,6 +292,17 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain *d,
>   	return 0;
>   }
>   
> +static void limbo_release_entry(struct rmid_entry *entry)
> +{
> +	lockdep_assert_held(&rdtgroup_mutex);
> +
> +	rmid_limbo_count--;
> +	list_add_tail(&entry->list, &rmid_free_lru);
> +
> +	if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID))
> +		closid_num_dirty_rmid[entry->closid]--;


Maybe define some helpers (along with other similar ones) in resctrl.h 
like this:

#ifdef CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID
static inline void closid_num_dirty_rmid_dec(struct rmid_entry *entry)
{
		closid_num_dirty_rmid[entry->closid]--;
}
...
#else
static inline void closid_num_dirty_rmid_dec(struct rmid_entry *unused)
{
}
...
#endif

Then directly call the helper here:

+		closid_num_dirty_rmid_dec(entry);

On x86 this is noop without occupy any space and cleaner code.

> +}
> +
>   /*
>    * Check the RMIDs that are marked as busy for this domain. If the
>    * reported LLC occupancy is below the threshold clear the busy bit and
> @@ -321,10 +339,8 @@ void __check_limbo(struct rdt_domain *d, bool force_free)
>   
>   		if (force_free || !rmid_dirty) {
>   			clear_bit(idx, d->rmid_busy_llc);
> -			if (!--entry->busy) {
> -				rmid_limbo_count--;
> -				list_add_tail(&entry->list, &rmid_free_lru);
> -			}
> +			if (!--entry->busy)
> +				limbo_release_entry(entry);
>   		}
>   		cur_idx = idx + 1;
>   	}
> @@ -391,6 +407,8 @@ static void add_rmid_to_limbo(struct rmid_entry *entry)
>   	u64 val = 0;
>   	u32 idx;
>   
> +	lockdep_assert_held(&rdtgroup_mutex);
> +
>   	idx = resctrl_arch_rmid_idx_encode(entry->closid, entry->rmid);
>   
>   	entry->busy = 0;
> @@ -416,9 +434,11 @@ static void add_rmid_to_limbo(struct rmid_entry *entry)
>   	}
>   	put_cpu();
>   
> -	if (entry->busy)
> +	if (entry->busy) {
>   		rmid_limbo_count++;
> -	else
> +		if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID))
> +			closid_num_dirty_rmid[entry->closid]++;

Ditto.

> +	} else
>   		list_add_tail(&entry->list, &rmid_free_lru);
>   }
>   
> @@ -782,13 +802,28 @@ void mbm_setup_overflow_handler(struct rdt_domain *dom, unsigned long delay_ms)
>   static int dom_data_init(struct rdt_resource *r)
>   {
>   	u32 idx_limit = resctrl_arch_system_num_rmid_idx();
> +	u32 num_closid = resctrl_arch_get_num_closid(r);
>   	struct rmid_entry *entry = NULL;
>   	u32 idx;
>   	int i;
>   
> +	if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
> +		int *tmp;
> +
> +		tmp = kcalloc(num_closid, sizeof(int), GFP_KERNEL);
> +		if (!tmp)
> +			return -ENOMEM;
> +
> +		mutex_lock(&rdtgroup_mutex);

data_init() is called in __init. No need to lock here, right?

> +		closid_num_dirty_rmid = tmp;
> +		mutex_unlock(&rdtgroup_mutex);
> +	}
> +

This code is also can be defined as a helper in resctrl.h.

>   	rmid_ptrs = kcalloc(idx_limit, sizeof(struct rmid_entry), GFP_KERNEL);
> -	if (!rmid_ptrs)
> +	if (!rmid_ptrs) {
> +		kfree(closid_num_dirty_rmid);
>   		return -ENOMEM;
> +	}
>   
>   	for (i = 0; i < idx_limit; i++) {
>   		entry = &rmid_ptrs[i];

Thanks.

-Fenghua