Received-SPF: pass (google.com: domain of linux-kernel+bounces-186723-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <7571640a-2250-4e35-aeab-d25cacf2135b@intel.com>
Date: Wed, 22 May 2024 14:13:26 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v18 08/17] x86/resctrl: Add and initialize display_id
 field to struct rdt_mon_domain
To: Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>,
	"Maciej Wieczor-Retman" <maciej.wieczor-retman@intel.com>, Peter Newman
	<peternewman@google.com>, James Morse <james.morse@arm.com>, Babu Moger
	<babu.moger@amd.com>, Drew Fustini <dfustini@baylibre.com>, Dave Martin
	<Dave.Martin@arm.com>
CC: <x86@kernel.org>, <linux-kernel@vger.kernel.org>,
	<patches@lists.linux.dev>
References: <20240515222326.74166-1-tony.luck@intel.com>
 <20240515222326.74166-9-tony.luck@intel.com>
Content-Language: en-US
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20240515222326.74166-9-tony.luck@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bVZWUUpFTzBQUG01Umt2bE0vWExGbTBDSzJoUEMzRnJ0WFY1d2p5SHArNHJS?=
 =?utf-8?B?RGJGaUFGNjA1UzZmVEdiV1d4WDluQWVISEZ1NEg0SW1FUEx4My9rYjE4bTZT?=
 =?utf-8?B?ZjFrbTJJSldVVmhZM3pVcTg1T0prakNCQnJnZHM2VHhuMUJoalY4bCtkbGoz?=
 =?utf-8?B?bjk2Z0JyQzhMOUNjWk9HSlc3TUZiUzMzVkFWbzRIL2pOZGZDQjlGWEJaUUpi?=
 =?utf-8?B?R2hLOFhpMjFxL2ppbHBvY2pCVjY0NmYzYUpKeFhMTC9xeHNaTEc2dS9aK3hW?=
 =?utf-8?B?R1ptb2xvVEJQemw5dlhVNkdlSlNUcmpDU1c3NGVCckJrOXFrVWxWWFlyOXhC?=
 =?utf-8?B?cDVNYUlYNyszcVlYckRLS0h3SjhBUmNZNE9VM0VuYWpIdUNOdXNQbnkvcFcx?=
 =?utf-8?B?eFFvR2U1dCt5ME13YXpreEVnNU4rSWprUUFoWDhvSnFZcTBGMmZxVHZna3Fp?=
 =?utf-8?B?azgySzdUZlBRWUZ1NHU0QzQ3dDRDcGExdnBZK2E0cktCVTI4U2h6Qm1qRyti?=
 =?utf-8?B?NlVIZEJkUnE2QmUwQkNGcm5ETjZjOUFHTzNYeHNjTmtHNjk0TmUzUGdtUEF2?=
 =?utf-8?B?YTBTZXkxUFVQMWJQZU51SS9mcFJUWGdUVXEyTFZVaFlDcFZ3MGtDL1lPb0tO?=
 =?utf-8?B?VUgySDMzS0FrZnI3NWZpa3RZNHNFUVNQd2RiZm02M2VFUTREMUd4ZUkxRDJ2?=
 =?utf-8?B?RnJMc1F1VUJ6L0xkanRFTDlWU0JQaWh2ajFCVG0wMURTdlNDRStHMDNzVk5V?=
 =?utf-8?B?K3QvcmdSVEtkQklmRXlNZkYrTXcxcTBjWFNUcThKKzZWTVMrcHEvNEN2N29Q?=
 =?utf-8?B?YnplM3R0Vzh3cVJhdURicFl0M1psck5BSTdPR052c01HUUVHZHNiRWU0Ty9J?=
 =?utf-8?B?bXRqZ3MwWmZVVGUwMlR2RzhGTktJV0xqVmFHVW14N1ZlemRmeU5IcUUzWGRq?=
 =?utf-8?B?TVRxWlRyRmlEdDVkV2JDdHhFTktuWHdVRFZnNVlJSXozeHl4TmVjQUpQZ01Q?=
 =?utf-8?B?RmV4T2J0Yll5eXpEMHhEREcyV3pQc244bTdnSXY4eVFHMlBBY3ppNE14YzR1?=
 =?utf-8?B?SzdIOXQwWFFyQzBodzlMMXJZV3QyOXJiejFwT1JFV3ZpaFNPNmNzVm1Od09X?=
 =?utf-8?B?bklNT2R5VEprQkhEL0E3bUJrZmRzWndUd3RzK0lmZVI1UlBiNk1aL1F5MkZl?=
 =?utf-8?B?LzNaU0M2eklMcWVyZHU3Z1Q0cHpsK0dSZmVEVjZMSyt6WG5QN2ZWZjlQSitD?=
 =?utf-8?B?Q3hYdWhpV2EvWWdsTS9MOER0VVdGMWJrY0dxbm1zTVRqT2RDbjFySlQ5ckp1?=
 =?utf-8?B?QUxsTkdqR1NUeTVHNHFiSFA3SHRSNkhEQnpkTFR3MFQra0YvVWc0VGpPc1c2?=
 =?utf-8?B?RytGYmhOeWVqZ0ZVWmlQNi9tNUhGZXVGQnBWQzViOWhhUkNjMVJuTVAwem94?=
 =?utf-8?B?bmhIaTBwN2ZvSkxLaDlGT0RCMitXUFlZTGQxVGl5czZqVUh0RmhwR2VwQ3JD?=
 =?utf-8?B?cW9jd3kyTm5QYnh6aEJSSUhMbkduRTcvSDhUU3VPWmlQN2FpbllSbm1jd1ZG?=
 =?utf-8?B?VmpidzhLb2lBRkhJb01nUmxnWDRGNlRPYUlWYlV2aXNySFVPeGRtYXgwYlZ1?=
 =?utf-8?B?aiswckxoTUhjdlJCZGRodTU2SjJROFp2TldVWldodjZkdkpMY21FMWlMa3RO?=
 =?utf-8?B?VWw2ZjN6cFVvVkdPeXYvSTcyZzlkVjFLbXRSVGRETFd5TEUyVVRZRVFVTEti?=
 =?utf-8?B?eGZKY1VuWUZEeXJzc2V4S0EvVmhyTUZpcmJxMXJva1BkWWRZdWZZM1VLWnlU?=
 =?utf-8?B?ZEFBWktzZWYxemwzSXlPMWROK2J4a25QbFRQVHErQmczb2Fyc012YjBRajJ3?=
 =?utf-8?B?dUdMdDRzTy8wemRXWjYvd1lkcUVUcGZUQWgrd0g1MGExelF3Nm5malJMN3Vx?=
 =?utf-8?B?UjQreUFxVHJKN3IzR1BuaURxQm5ETlVDVGRxWFpXa0FKV1hodEhMVkdTdVA4?=
 =?utf-8?B?cWtlRXRIMUx2WkhROVdXVHVGd01nVFpwU2F3YmVuKzMrUGxKRGJudlJKZ09U?=
 =?utf-8?B?dEk3b0RaSzBuUWlJMjlyb0J4NjZkVnFmS1lhWHlWZWVsSUZPNkx0L2JtaENa?=
 =?utf-8?B?WC9va3hqZU9GTE5lMmR4bitTb0g2RjdZQmY2d1FiWU43SjdRUm1Rd1dFSWVw?=
 =?utf-8?B?L2c9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 0d4ddc52-c324-4b2d-94c6-08dc7aa40657
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 May 2024 21:13:28.0482
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ftEGa6cL3sQJ2uneCwa8gAiFUyb7+Ml423cf2rmJiUQcPpS/e34kvxsTZB+RZA4dWNDbt0W32mHwwrA7S3uJMvlEtHlZ9JjvlARXFEBF+to=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR11MB7671
X-OriginatorOrg: intel.com

Hi Tony,

On 5/15/2024 3:23 PM, Tony Luck wrote:
> When Sub-NUMA (SNC) mode is enabled monitoring domains are created at

Sub-NUMA Cluster (SNC) ?

> SNC node scope. Add a field that holds the identity of the L3 cache for

This is not necessarily the L3 cache, but instead intended to be the
monitoring display scope, no?

> each domain to make it easy to find all domains that share the same
> L3 cache instance. There are three places where this is needed. In
> all cases code is operating on a domain where "d->id" refers to the
> SNC node id.
> 
> 1) When making monitor directories.
>    Need the L3 cache instance ID to make the mon_L3_XX directory
>    that will contain the legacy monitor reporting files and the
>    mon_sub_L3_YY directory for this domain.
> 2) When removing monitor directories.
>    Similar to making directories.
> 3) When reporting data from one of the L3-scoped legacy files.
>    This requires summing data from each SNC node that shares the
>    same L3 cache instance id.

<insert motivation about why this cannot be determined dynamically
at the places identified>

> 
> Signed-off-by: Tony Luck <tony.luck@intel.com>
> ---
>  include/linux/resctrl.h            | 2 ++
>  arch/x86/kernel/cpu/resctrl/core.c | 8 ++++++++
>  2 files changed, 10 insertions(+)
> 
> diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
> index 98c0ff8ba005..2f8ac925bc18 100644
> --- a/include/linux/resctrl.h
> +++ b/include/linux/resctrl.h
> @@ -96,6 +96,7 @@ struct rdt_ctrl_domain {
>  /**
>   * struct rdt_mon_domain - group of CPUs sharing a resctrl monitor resource
>   * @hdr:		common header for different domain types
> + * @display_id:		shared id used to identify domains to be summed for display

This description seems to indicate this is a member only used when needing to
sum domains, thus only for SNC at this time. Looking ahead the description does not
seem to capture that this value has been integrated into non-SNC support and will
always be used when creating files for all domains, whether SNC is enabled or not.
This member thus seems to be used for more than it claims to.

>   * @rmid_busy_llc:	bitmap of which limbo RMIDs are above threshold
>   * @mbm_total:		saved state for MBM total bandwidth
>   * @mbm_local:		saved state for MBM local bandwidth
> @@ -106,6 +107,7 @@ struct rdt_ctrl_domain {
>   */
>  struct rdt_mon_domain {
>  	struct rdt_domain_hdr		hdr;
> +	int				display_id;
>  	unsigned long			*rmid_busy_llc;
>  	struct mbm_state		*mbm_total;
>  	struct mbm_state		*mbm_local;
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 15856254fea7..dd40c998df72 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -614,6 +614,14 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
>  
>  	d = &hw_dom->d_resctrl;
>  	d->hdr.id = id;
> +	d->display_id = get_domain_id_from_scope(cpu, r->mon_display_scope);
> +	if (d->display_id < 0) {
> +		pr_warn_once("Can't find monitor domain display id for CPU:%d scope:%d for resource %s\n",
> +			     cpu, r->mon_display_scope, r->name);
> +		mon_domain_free(hw_dom);
> +		return;
> +	}
> +
>  	d->hdr.type = RESCTRL_MON_DOMAIN;
>  	cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
>  

Reinette