Received-SPF: pass (google.com: domain of linux-kernel+bounces-79044-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <371242fb-dce2-4de0-bba9-4d85475d5d9a@intel.com>
Date: Fri, 23 Feb 2024 11:41:48 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v2 2/2] x86/resctrl: Add tracepoint for llc_occupancy
 tracking
Content-Language: en-US
To: Haifeng Xu <haifeng.xu@shopee.com>, James Morse <james.morse@arm.com>
CC: <fenghua.yu@intel.com>, <babu.moger@amd.com>, <peternewman@google.com>,
	<x86@kernel.org>, <linux-kernel@vger.kernel.org>
References: <20240221092101.90740-1-haifeng.xu@shopee.com>
 <20240221092101.90740-3-haifeng.xu@shopee.com>
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20240221092101.90740-3-haifeng.xu@shopee.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YjF3cnVhWFJzT2QxZ1hKTTlmU2dUR05QWnJVMDBCejBoMGhxSGlvU08xdVlR?=
 =?utf-8?B?UXhhZmV6M0txdHRybTBBckRRc3YzNmN3OWh0S1lwYTZkWENaVUExTzVzSkVo?=
 =?utf-8?B?UE85RlN0LzA0aiswL09xaC8rUWM3L09uYk9rVjE5eGxqOWdXTnhaRHVmQXMr?=
 =?utf-8?B?dXlSbEQwNzU4cmdHR21WU01TQ2kvZnh4WE53eFIyWDluTXBISHc0aU9wOXVS?=
 =?utf-8?B?VW1rRUlOUW5XQW1IVGQwamtubTVLL1VTanYzZkpSY0ZETmU4Uk9JbnJQYWNL?=
 =?utf-8?B?UW1PZXFySURoOTBSVWVhc2VpUVZKOGdYdzNsekVVNWNTcVIwUWpJckJMeExM?=
 =?utf-8?B?RldIbFJkMmo3aGRNaHhmUjhabUNHd0JKVGp5aGh0OGM3Sk43SE5PUHlBOVFw?=
 =?utf-8?B?Mys5eWR1UlhNOGhyK2VGYmlya0ZUc2RkZFhDRm1FaUZqTDdub0JUUm8zZ1VS?=
 =?utf-8?B?akQ2ajJzRnF4ZXhDNlFER1I3QzJsWHl0SldDdEwvRDhva01Jeisvc21tU2pM?=
 =?utf-8?B?TnpWY2pFdmpoeEZOUXV2MnQzQVhqYWlSSk5qQ0EyTzltSjZoVWh3ZUduUHdk?=
 =?utf-8?B?WlVhRk1DQm1BeFdmbXJsY2M4NHhJL2xDejQ2SVJYVFZTZ3JNU29Wci9ncjMz?=
 =?utf-8?B?aVpXTWlzL1RCeUZFM0UrSEJFMk9LNUdMYk82Ym9KWSszd2QrTldXdG1MVjVC?=
 =?utf-8?B?RUNrNDBtRmV3b0dSTm5Fb3laTXY1YVNmMkxzL1oyaTcrZnJZUHVDTVg4TUE0?=
 =?utf-8?B?NHUzQWo2blh0R0RPV2t1eG03V1p5WjUzZExheTNWSWVVZUo5ZjFldElXeHVU?=
 =?utf-8?B?OE5KaEprdFBDa2tETVA2QURDYnduMmpnR211Sjd4Q1Q0NDYzc3lObkpxVnNi?=
 =?utf-8?B?WjZORWRHN09uV1Y0YktWdEhSZ0hqL1lZQm9CU2RObW9zWE9kUG1VWndFMVNU?=
 =?utf-8?B?ODJHakpjbU52SHE0UHdCcVZXdDNGNDcwVktUN1ZSRnN1MkNhbjJxV2EvK29p?=
 =?utf-8?B?UGRkSEhJcDNwemI5bVlzUzhlNkNyZC82VXNqcERFVjZsbE00OUx2ZXhDWHRK?=
 =?utf-8?B?THFYQUovRTc4V2p3dXZxMjJuWkw4bE8wOXU2QUN3aW1hMjNkY0hvUUNnTkpX?=
 =?utf-8?B?MHNxWG1SUjFSL1J5MldFRGNCcjArRTlmRjhLckZOSnVtQ3BZc1BxTk5Md0Ur?=
 =?utf-8?B?d2N2aTBXaTVKeXhwcmhYcUF6a2pQUlNIekdFRUpwUytMYmdqeVFmZ3ZtVGk0?=
 =?utf-8?B?NVhyMzBPRjRxNkFzUEhBbWVFYXpJY2FpL3dOck5rU1dFR0szUTFlN0lJUHh5?=
 =?utf-8?B?MG4xL1ZwUUQvVHk0Z0ZUWlpJaTV4Y053dWphTDljeWpOR2I3YkZDTkYzS0FR?=
 =?utf-8?B?d2QwYzBhRmJhRVNEMGtZamc0ekVwSGVvS0FWZmtWemlCM0FobXNZeW9CV1FI?=
 =?utf-8?B?NUNpNjBxcGd2UW5BRHo3bnZaY2ZCcEthSjJPdmh6eEQrZDNRTVFkd0xNTkNt?=
 =?utf-8?B?ck9yODFyRWw2cEVKQTNOa1RKWFBLVURXWmQrbElPK3NNNWRONmFBRXl3TzNz?=
 =?utf-8?B?NTZ4NVJZQmJiQmpHYWhmRjdDWXNNRkg1UWNPM25tTHhXL3RNSTk4bTFxeC9o?=
 =?utf-8?B?NjkydkdlRDNSaytic2tFem16TEFCa2FRYVRFSEFzTXkzRGFzQ0xaZ3hlZUQ4?=
 =?utf-8?B?QzVCaGg1NWd2OUJsclo0dFlhd0tITEVKQnQzK1dycnplMExRYnMrNFdaRW5Q?=
 =?utf-8?B?bTNUWlI3cUV4ZXpOM3Y3V1o4RWpheEFtYitieDB5MFFDYmVsMHlzVnJ3K3lu?=
 =?utf-8?B?bVlicitLVTVwVEVkQW1ORTgzVGdqOHAySXNXVVcxUjNJWit2TVJOU0ppc2Jn?=
 =?utf-8?B?c3BTSFRwYVN2eXYvSUtWb3hHdjV3Y0NFVlIyMjRjbjRnc3pCSENBdGQzU3dB?=
 =?utf-8?B?WkNXUE1CdG51MFhrdkVHRWNVd2p2NFE0Y25vbkMzU2lOR3RRTmY0NEZ0OHRO?=
 =?utf-8?B?d0JhUk5RNHJrM2FKRFJWWW1Ldmk2amNHbTgwTTZWZktXMDJnKzFTc3BIQTY4?=
 =?utf-8?B?L1A0VVkvLzVHZjdoMzZET3cycENOcW5WQzYrS3AvaG5tVmxRbVRjYzFUM29s?=
 =?utf-8?B?d1Zpd1p1eTZuZG1HMjBkc2JpcUp4L2NrbTFjSklqL0FXYlJjSUdxd1pEdUpw?=
 =?utf-8?B?c3c9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 644cea91-9081-4291-780b-08dc34a77ad6
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2024 19:41:50.6380
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mVm+7HzckROd0V3nyJpNNHd1wJJSacUtBYUcaAyVMiTBZFpfJIBU4d//f5GIJg0/dmzePkYMgpG6IK2BGZP9ieWH24BV4FglUyNjW42gYTY=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB6881
X-OriginatorOrg: intel.com

(+James)

Hi Haifeng and James,

On 2/21/2024 1:21 AM, Haifeng Xu wrote:
> In our production environment, after removing monitor groups, those unused
> RMIDs get stuck in the limbo list forever because their llc_occupancy are
> always larger than the threshold. But the unused RMIDs can be successfully
> freed by turning up the threshold.
> 
> In order to know how much the threshold should be, the following steps can
> be taken to acquire the llc_occupancy of RMIDs in each rdt domain:
> 
> 1) perf probe -a '__rmid_read eventid rmid'
>    perf probe -a '__rmid_read%return $retval'
> 2) perf record -e probe:__rmid_read -e probe:__rmid_read__return -aR sleep 10
> 3) perf script > __rmid_read.txt
> 4) cat __rmid_read.txt | grep "eventid=0x1 " -A 1 | grep "kworker" > llc_occupnacy.txt
> 

The details on how perf can be used was useful during the discussion of this
work but can be omitted from this changelog.

> Instead of using perf tool to track llc_occupancy and filter the log manually,
> it is more convenient for users to use tracepoint to do this work. So add a new
> tracepoint that shows the llc_occupancy of busy RMIDs when scanning the limbo
> list.
> 
> Signed-off-by: Haifeng Xu <haifeng.xu@shopee.com>
> Suggested-by: Reinette Chatre <reinette.chatre@intel.com>
> ---
>  arch/x86/kernel/cpu/resctrl/monitor.c |  2 ++
>  arch/x86/kernel/cpu/resctrl/trace.h   | 13 +++++++++++++
>  2 files changed, 15 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
> index f136ac046851..1533b1932b49 100644
> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
> @@ -23,6 +23,7 @@
>  #include <asm/resctrl.h>
>  
>  #include "internal.h"
> +#include "trace.h"
>  
>  struct rmid_entry {
>  	u32				rmid;
> @@ -302,6 +303,7 @@ void __check_limbo(struct rdt_domain *d, bool force_free)
>  			}
>  		}
>  		crmid = nrmid + 1;
> +		trace_mon_llc_occupancy_limbo(nrmid, d->id, val);

This area recently received some changes (you can find the latest on the
x86/cache branch of the tip repo). Please see [1] for a good
description of the new "index". For this tracing to be useful to MPAM
I thus expect that the tracepoint will need to print the MPAM equivalent
to CLOSID, the PARTID. We can refer to this CLOSID/PARTID value as
"ctrl_hw_id".

This snippet can then change to use the new resctrl_arch_rmid_idx_decode()
to learn the "ctrl_hw_id" and "mon_hw_id" and print it as part of
tracepoint:
"ctrl_hw_id=%u mon_hw_id=%u domain=%d llc_occupancy=%llu"

This will be filesystem code so it cannot know how an architecture
treats these numbers. Consequently, this may look strange to x86 users
when ctrl_hw_id will always be X86_RESCTRL_EMPTY_CLOSID ... but it should
be clear that it is invalid? 

James, what do you think? Any thoughts on how MPAM will use the limbo handler
to understand what information can be useful to the user here?

Reinette

[1] https://lore.kernel.org/lkml/20240213184438.16675-7-james.morse@arm.com/