Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <769521ee-6907-1613-442f-e2915a8ea554@intel.com>
Date:   Tue, 18 Jul 2023 13:40:32 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.13.0
Subject: Re: [PATCH v3 3/8] x86/resctrl: Add a new node-scoped resource to
 rdt_resources_all[]
Content-Language: en-US
To:     Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>,
        "Peter Newman" <peternewman@google.com>,
        Jonathan Corbet <corbet@lwn.net>,
        "Shuah Khan" <skhan@linuxfoundation.org>, <x86@kernel.org>
CC:     Shaopeng Tan <tan.shaopeng@fujitsu.com>,
        James Morse <james.morse@arm.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Babu Moger <babu.moger@amd.com>,
        Randy Dunlap <rdunlap@infradead.org>,
        <linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
        <patches@lists.linux.dev>
References: <20230713163207.219710-1-tony.luck@intel.com>
 <20230713163207.219710-4-tony.luck@intel.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20230713163207.219710-4-tony.luck@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TE80R2VzOHhJT0hoWXJRSXhlMXFkRUovYkw4WjM5aFF3Y0pHaXFHRWxoZ3hR?=
 =?utf-8?B?YnV0OXBvVzU3bms2eVV1RjRFdndKZnNmYU1xNnllSS84eXpPRmhKY042d3FE?=
 =?utf-8?B?WEtKMFVmMmUxQ09lVXdUMGk4MnZFV2RmUUpVeXNPSWhXY0wvS2VYb3E2UHIr?=
 =?utf-8?B?RzNBb3NpZ2Q4aFVsMnprbnc0bWVFZDJRRFgvdVU5VjljVGNUK0lDRmM4RzdL?=
 =?utf-8?B?ZWQ5LzBWQVIyU0J4REtYZkNiZ3dnRFB5SVFZd0pLUWEwV2w2RE41WHprWUEz?=
 =?utf-8?B?ZHZLcFFwTUd0dTlHZnJHZURMTFZBV2NrSkRvZmZISERFSktUUDdBS3lBcVdo?=
 =?utf-8?B?SzdHbGd1RkVtRnVpaXBpN3hTQldCR2s5OXJZZHorR3RPWmdGc1JpclJPY05Z?=
 =?utf-8?B?alVQaW1xWk1DaDlDTzVnUlJwVnN4MndKL3JqVDJvVXNkdHVPRXpJdGhOLzRs?=
 =?utf-8?B?dC81L2w5YjVMUGQ2Ym5zWDhSa3BSMFNwSGtoSWc2WXRxb1M4V21lUVJQb1hi?=
 =?utf-8?B?WjcxU1p2a0hudWc2YkQ5WUUzZDUwR1Z1eWJuMDJQcytFaFB3UWxoSnB6Q0NO?=
 =?utf-8?B?UGJva1FpQ1RQNW9LYmFVOFdwZ1NNbUJqdHF4REhsTjBPR2dSZnNFT1phZG1W?=
 =?utf-8?B?OXFvdWpTTnd5OTVVQkVOeFlVeUhFUjA1czdhTnNGendpR3J2TWR3VXhWVldy?=
 =?utf-8?B?RXVIaGxtNkJrYllaT0M3cTh0MTh3ZXhYTk1IOTNlQmxQdUhhOXAvZXQ5RmhY?=
 =?utf-8?B?TDVib3BzNG1XbDRrYzh1Y3A3ejRTMkdjUDZCdWplSGJId2FrU3M3RGNyekh4?=
 =?utf-8?B?TnpKNzdRNDNMdW05dkVpQVh5bm9FcHF6M1lTWFVzWlV4aVlQUHJYOHNZQ1h5?=
 =?utf-8?B?c0l6MXcrQUhFQ0M2cUJ4NTRuMjNkRkRLdm1MNFdVTUV2QnZ6bWcxczV3c1ZT?=
 =?utf-8?B?cWxiUjY4SERGQUVvUlY0cDJyRVlKQktjWlRUaDdsS3hxTDg2WVd4TWh2MUZa?=
 =?utf-8?B?aDdtZUZnanB1Z3J3UUI3SG84dmRLYzI1TWVvK0ZhV0ZkaUVjbnVrc015WU5T?=
 =?utf-8?B?aVJsUWxhQW5DQ3A0RStvL05WN1ZWcFhHQ21BbmlwYnJobExGSHBjSVp2Zk1h?=
 =?utf-8?B?czNJdyswQ2JwSnlHYnVpRnZmYmV5dFRDcjh5YWNraG1xTytNbW9HdisxV0Jm?=
 =?utf-8?B?cVVwSkVDQldFWUVhcG1TV2I5YnVkSzFNRC9CR0ZqM2lIckR6RGphY0R3UzY1?=
 =?utf-8?B?ZzJ1c0FTSWhiM2tnMlZZWGZ3N0lJSXFoVmU4ak12TkpYaGpGRGJ3Y042R1FV?=
 =?utf-8?B?cldOMEx1aGwvTUFiTG1qYkhDWTlEZndSdzFaTmoydWtrQnBSSEpyVHBGK2cx?=
 =?utf-8?B?aTNucnppeFN2dUg5RDdCVU4raHpOWENDZ2N3dHJHL0ZveVFkM1pXTTFETHA5?=
 =?utf-8?B?UlgyT0tGZHBEQ1FnaUFjVXZoUmE3Tytwb1JUa0xiNmYzMWJJcHpDbEFrbklO?=
 =?utf-8?B?anFUWGpqRWlVMndHZzd6dTJCSGlFaVRDYitGL1NIR2t0cERmSjBLeWVmS1ov?=
 =?utf-8?B?S3hFeVVQTFNYOHMwRlFwQjRZNHMyOExqWTQ2azJ6Z3dmTDdXeU4rQVhMT2Fy?=
 =?utf-8?B?NVJYbUFITzEwMmVWVXhWTGdLT3BZQU9SRmY1SklXZ295UWlyRkEwQ2RLR3VL?=
 =?utf-8?B?ZXIyRGtER2RzL2JLN2drZ3pWS2lveE11aFJmY0tVaW54ODNja2ZtYmVOOHdk?=
 =?utf-8?B?eFZIQ0xCZ0Y3UzNFK2E5endrUFlpRkdETHgzdmliRnh3MWxQeDBCUi9GSnVk?=
 =?utf-8?B?eFcwTVdURklvZ2ZVTHJaR1dCdXBMS3ViL0srU2RmUkRoVEIyWmpDeE1pYWdY?=
 =?utf-8?B?am9LVjd1c0pXT2NKMlhqVGxtM1JwdGN0VXZSK1k1RDdabHpWTkd6M05tNlRu?=
 =?utf-8?B?MllnWTdJZmVSWGpQbUNtZFRDeS84MW95TFNEM1YrSG5ZNHROSEJzVS9xWFoz?=
 =?utf-8?B?c1FqMGxNbWR1ekptLzd5M2hHNXU0MmxPVTdzWjhoY0ltTlQxblVsa2lQTmRL?=
 =?utf-8?B?dElWUTMwaWRNWm45THhFRjJmRGY4SThFY1lkVjRXOXErbWdwN0MwSzJjZVQx?=
 =?utf-8?B?cnljVmI5clJJMnIrcDlUdWpUSWVJWXR3dnI1bWlQZm51K3pzLzdCbkdoMmJR?=
 =?utf-8?B?dkE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1ac4d4e7-35b9-458b-3a38-08db87cf3ccc
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2023 20:40:35.2343
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pHBSXRP84Uuwt2bJRSB/OmyU84hQRDxhuxm+zJ9K3yKX03uTvfXd3bOTwQ+Qxqg8exxt18KTrth4r0dD1zOY3MJQiZH9zPHt4h5n8DD+71Q=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7890
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Tony,

On 7/13/2023 9:32 AM, Tony Luck wrote:
> Add a placeholder in the array of struct rdt_hw_resource to be used
> for event monitoring of systems with Sub-NUMA Cluster enabled.

Could you please elaborate why a new resource is required?


> 
> Update get_domain_id() to handle SCOPE_NODE.
> 
> Signed-off-by: Tony Luck <tony.luck@intel.com>
> Reviewed-by: Peter Newman <peternewman@google.com>
> ---
>  arch/x86/kernel/cpu/resctrl/internal.h |  4 +++-
>  arch/x86/kernel/cpu/resctrl/core.c     | 12 ++++++++++++
>  2 files changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
> index 8275b8a74f7e..243017096ddf 100644
> --- a/arch/x86/kernel/cpu/resctrl/internal.h
> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
> @@ -435,6 +435,7 @@ enum resctrl_res_level {
>  	RDT_RESOURCE_L2,
>  	RDT_RESOURCE_MBA,
>  	RDT_RESOURCE_SMBA,
> +	RDT_RESOURCE_NODE,
>  
>  	/* Must be the last */
>  	RDT_NUM_RESOURCES,
> @@ -442,7 +443,8 @@ enum resctrl_res_level {
>  
>  enum resctrl_scope {
>  	SCOPE_L2_CACHE = 2,
> -	SCOPE_L3_CACHE = 3
> +	SCOPE_L3_CACHE = 3,
> +	SCOPE_NODE,
>  };

A new resource _and_ a new scope is added. Could changelog please
explain why this is required?

>  
>  static inline struct rdt_resource *resctrl_inc(struct rdt_resource *res)
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 6571514752f3..e4bd3072927c 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -112,6 +112,16 @@ struct rdt_hw_resource rdt_resources_all[] = {
>  			.fflags			= RFTYPE_RES_MB,
>  		},
>  	},
> +	[RDT_RESOURCE_NODE] =
> +	{
> +		.r_resctrl = {
> +			.rid			= RDT_RESOURCE_NODE,
> +			.name			= "L3",
> +			.scope			= SCOPE_NODE,
> +			.domains		= domain_init(RDT_RESOURCE_NODE),
> +			.fflags			= 0,
> +		},
> +	},
>  };

So the new resource has the same name, from user perspective,
as RDT_RESOURCE_L3. From this perspective it thus seems to be a
shadow of RDT_RESOURCE_L3 that is used as alternative for some properties
of the actual RDT_RESOURCE_L3? This is starting to look as though this
solution is wrenching itself into current architecture.

From what I can tell the monitoring in SNC environment needs a different
domain list because of the change in scope. What else is needed in the
resource that is different from the existing L3 resource? Could the
monitoring scope of a resource not instead be made distinct from its
allocation scope? By default monitoring and allocation scope will be
the same and thus use the same domain list but when SNC is enabled
then monitoring uses a different domain list.
  
>  /*
> @@ -489,6 +499,8 @@ static int arch_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_domain *hw_dom)
>  
>  static int get_domain_id(int cpu, enum resctrl_scope scope)
>  {
> +	if (scope == SCOPE_NODE)
> +		return cpu_to_node(cpu);
>  	return get_cpu_cacheinfo_id(cpu, scope);
>  }
>  

Reinette