Received-SPF: pass (google.com: domain of linux-kernel+bounces-37737-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <8abb0237-90e3-484e-a549-d74f540d4045@intel.com>
Date: Wed, 24 Jan 2024 14:25:09 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 3/3] x86/resctrl: Display cache occupancy of busy RMIDs
Content-Language: en-US
To: Haifeng Xu <haifeng.xu@shopee.com>
CC: <fenghua.yu@intel.com>, <babu.moger@amd.com>, <peternewman@google.com>,
	<x86@kernel.org>, <linux-kernel@vger.kernel.org>, James Morse
	<james.morse@arm.com>
References: <20240123092024.1271882-3-haifeng.xu@shopee.com>
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20240123092024.1271882-3-haifeng.xu@shopee.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cHdYUXdhazZLM3RmZllCZk1VNXB4dU5KOEdYVGZ3NkRJeGFVQ0VUUlBHTTkr?=
 =?utf-8?B?SnE5TDhxRkR2OUtIL3NtOWhTb0d1R1ZRYVlLNktBOURQRWxoK2ZCdlV2Mkw1?=
 =?utf-8?B?RFMxZ0hsRk9YSXVEVWJqbExjdUorNjljVk1NeEdJRVJpVi9HVFY3QmVqa0Vq?=
 =?utf-8?B?WDdJWUFKR2pmemUxajNMSlFFNjF0a2hHSEN0MG5GQmRuTzVPb29nQXk0WkNG?=
 =?utf-8?B?YmpOMVNTdG1HbGw3SkVzd0NRcmFZUTVqeGozWlZyMm1rZDc1OXAyNjFhN2tF?=
 =?utf-8?B?VVo3WlJxN24xNVJrazhoUmlJclh4dDlOcnRyN3ZvdTl0b3p0cG9kbXo1bUJl?=
 =?utf-8?B?QmhiVU5nSzZPT2VHVURjczBRdHFaWEFqMlYvNE82MEcvbUlIeHdlKzFuRHdV?=
 =?utf-8?B?d1BuRURVTFduNVF3b2RsbVlVV2VUL1dmd1hCRFFsTmFPd3JLalBuK1ZWSTg1?=
 =?utf-8?B?dzJYUFJJZDJnbEVnRkJPdDUvV2txWkxBcjBodUQ5QUE4RTlRT2ZYSDVCWEc2?=
 =?utf-8?B?Vkh1Q256U0VaU25wYkgyeXFKQnVmYnpCaXNjZHZLSkovbTF2Q3BWMHJ0VEhm?=
 =?utf-8?B?bUhZckZMZFBBSjJkZDk4aVJxUi9LRVVtWFI2MmdWRitnYTJkNDdPUXJNOUps?=
 =?utf-8?B?QnFFOWlrMkdleUhuNnpGUEU0dml4VEY0UlV3KzRxNUZoUWNzTWhUMWZsMkJV?=
 =?utf-8?B?b2ZtQURLL1VrM3d4dVNoVWNQRmp6cHR3Y1kvNlRRNEhkeElqZ0Y4aW5Ud3My?=
 =?utf-8?B?cVdKL01DVzdJNHNaVHE1am5LR0U2YzlCVEg1aVFxbC9Ub25yc1kxM3M4dVhl?=
 =?utf-8?B?S0dOWDhwZnFSSzJkQXVsbkpkMWQ0VnkwVWgyVnFvQ0tBN3NuL1B4RGZJM3l1?=
 =?utf-8?B?dmtRRld4cm9reHgrUnB1Y1hiVGVMN1hQMDA2ZVpzclBqekxSQnJTMTJ5amdn?=
 =?utf-8?B?R0NKYzluYUxpVENXV1lURnZhMmR3NTNva3hRZGI2dmF1QTJHLzdaV3hDSTdo?=
 =?utf-8?B?dFhlUSsxUk84UWcrRjJZNUpMMFFPbXNvc0RvUXJWaHlHaWppY0tmL2l3U1E1?=
 =?utf-8?B?RkhVYlpRNlpzMzdLcmxPRVBETVBxa3F0dHZoNUZ0Wk1IQnV2dnRNZDRFekdm?=
 =?utf-8?B?QzR1SnBGWHluS0hYcUtSdHpYYmJpTWRQcEhrMnZuWGwxUmhRR1VNM1hOaUVu?=
 =?utf-8?B?dUcxaWNLZkNYM0E3cXE0dDdwUmxjNmpwckliREQ1UWdmZlovSERJRlBSRTlD?=
 =?utf-8?B?cGQxWWZ3c0wrNm9CZnFDWVg5clAvZm85UkYxbXphaFJiNGpIYlI3VGwwUldL?=
 =?utf-8?B?ZzY3MzJqVHlKNWtxaE83cWtHbGRySUVTUWdlc1lDejZhOGNBa0t0WHJXejNi?=
 =?utf-8?B?Q0d4elpVQ3NXdkd2QjBLNHVPWFQvR2dSUkM5OHVqcGpXd1ZWeTdTZnYvQjMy?=
 =?utf-8?B?WVpoaHVhdEJaa3RxTTIvR0Q0MkZvTUlLZlYzUEVqNWVCaUZrdUQ0N0RNWlJS?=
 =?utf-8?B?eHFWdFkyd2hNeWFBeTNadjRWYWxYRVd4OGRORzhkUHgybGNReGlpUzdzUUJ0?=
 =?utf-8?B?RWRtMldnOE56RFk0ZzZpem1rdFNEU0oyVjNNZGw3dkRyQXM0THNRVVFENExw?=
 =?utf-8?B?cjEvNnJTc0xNWFBHWitHYmlRbzcxRzBpek9VWmR0c1JYSCthZ3g5M3Q0SkR6?=
 =?utf-8?B?OEJ1WHkxd1EyZWE4aHltOFFmdHZVR3AvR1ZVVUxLWEhrelU5WS9oYmhxTWI3?=
 =?utf-8?B?TVpaK2lqbEk3SytBeldGY0FWWWMyWXg0UE1wM0cyMWEwS0w1WWJISFl5dWZi?=
 =?utf-8?B?bncxV3FhdndjZ0xmOE9QeFN4MzFFd0d3ZlljUHdsdERTdFF5RUlrQ20zU2Rl?=
 =?utf-8?B?cjRlejNORzdzQnZwLzRtTUQ2YzN1QVBCUVNvcTR0K1BSc3BMV0dYVnpxWGxY?=
 =?utf-8?B?eFBFVXdFTExNbG1qb2oyS1RYeWEyMUpVZzNraVhYZXpZZzlFRjlnUnhIeHJk?=
 =?utf-8?B?WEwwck92eFE3QVBXaUVxZFoydHc1bGtkVWFzNEd1Q0h0RnR1aGM0dE9sTkxu?=
 =?utf-8?B?ek9oWnRndFRuSzRsbGIrdldIdVUwc0tPUk1ERVlsUnhmWjVKZXNkdnNqZDBZ?=
 =?utf-8?B?MVE1bkVmUzRzMlVTMUUxOEpxaitzWVZxWEw0Nkxyak5zTWx1blRpVnlTUTYz?=
 =?utf-8?B?RXc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 8433d117-999c-460f-9243-08dc1d2b5372
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2024 22:25:10.1641
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: BIr3JNDTdy+1u5bq0g3QsApFYILNRAg/XgIgn28oZFVmvox81FrBv6WeLMp9fTEe71wcKiDRZX3MK4BKo4fdSqHqNh7vpaEc3nGntCt03dg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB5936
X-OriginatorOrg: intel.com

(+James)

Hi Haifeng,

On 1/23/2024 1:20 AM, Haifeng Xu wrote:
> If llc_occupany is enabled, the RMID may not be freed immediately unless
> its llc_occupany is less than the resctrl_rmid_realloc_threshold.
> 
> In our production environment, those unused RMIDs get stuck in the limbo
> list forever because their llc_occupancy are larger than the threshold.
> After turning it up , we can successfully free unused RMIDs and create
> new monitor groups. In order to accquire the llc_occupancy of RMIDs in
> each rdt domain, we use perf tool to track and filter the log manually.
> 
> It's not efficient enough. Therefore, we can add a RFTYPE_TOP_INFO file
> 'busy_rmids_info' that tells users the llc_occupancy of busy RMIDs. It
> can also help to guide users how much the resctrl_rmid_realloc_threshold
> should be.

I am addressing both patch 2/3 and patch 3/3 here.

First, please note that resctrl is obtaining support for Arm's Memory 
System Resource Partitioning and Monitoring (MPAM) and MPAM's monitoring
is done with a monitoring group that is dependent on the control group,
not independent as Intel and AMD. Please see [1] for more details.

resctrl is the generic interface that will be used to interact with RDT
on Intel, PQoS on AMD, and also MPAM on Arm. We thus need to ensure that
the interface is appropriate for all. Specifically, for Arm there is
no global "free RMID list", on Arm the free RMIDs (PMG in Arm language,
but rmid is the term that made it into resctrl) are per control group.

Second, this addition seems to be purely a debugging aid. I thus don't see
this as something that users may want/need all the time, yet when users do
want/need it, accurate data is preferred. To that end, the limbo
code already walks the busy list once per second. What if there is a
new tracepoint within the limbo code that shares the exact data used during
limbo list management? From what I can tell, this data, combined with the
per-monitor-group "mon_hw_id", should give user space sufficient data to
debug the scenarios mentioned in these patches.

I did add James to this discussion to make him aware of your requirements.
Please do include him in future submissions.

Reinette

[1] https://lore.kernel.org/all/20231215174343.13872-1-james.morse@arm.com/