Received-SPF: pass (google.com: domain of linux-kernel+bounces-108129-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <18ac2a1f-733c-4586-b9d8-d16bb9bfa2fe@intel.com>
Date: Tue, 19 Mar 2024 14:00:39 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v16 0/9] Add support for Sub-NUMA cluster (SNC) systems
Content-Language: en-US
To: "Luck, Tony" <tony.luck@intel.com>, "Yu, Fenghua" <fenghua.yu@intel.com>,
	Peter Newman <peternewman@google.com>, Jonathan Corbet <corbet@lwn.net>,
	Shuah Khan <skhan@linuxfoundation.org>, "x86@kernel.org" <x86@kernel.org>
CC: Shaopeng Tan <tan.shaopeng@fujitsu.com>, James Morse
	<james.morse@arm.com>, Jamie Iles <quic_jiles@quicinc.com>, Babu Moger
	<babu.moger@amd.com>, Randy Dunlap <rdunlap@infradead.org>, Drew Fustini
	<dfustini@baylibre.com>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "linux-doc@vger.kernel.org"
	<linux-doc@vger.kernel.org>, "patches@lists.linux.dev"
	<patches@lists.linux.dev>, "Wieczor-Retman, Maciej"
	<maciej.wieczor-retman@intel.com>
References: <20240312214247.91772-1-tony.luck@intel.com>
 <8ee6f553-16c0-4097-b5d8-af1598d1b85a@intel.com>
 <SJ1PR11MB6083B59F32F33C0FFBFCA1BCFC2C2@SJ1PR11MB6083.namprd11.prod.outlook.com>
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <SJ1PR11MB6083B59F32F33C0FFBFCA1BCFC2C2@SJ1PR11MB6083.namprd11.prod.outlook.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VXZKUjQ5Q0JkeExRNnRXTHlybC9pMmNwQW9LR2NseDloRHZFaWZzdlpBN1I5?=
 =?utf-8?B?VFlxR05tbGlxaDJnQkQxL3prZTVkbmMzdEx2NVRMM05wZHhoRWd1TW5Rd2d0?=
 =?utf-8?B?eFJEczRkdFRibDBycUEzYTZ3OEFxRXRLVWQ4OTdoSnduUUlFZUFsdjZ4bHND?=
 =?utf-8?B?M1NIa2VWYXFOTlAyR08rZ1UwbGNwNVJYNVJkNWdFVHRxQkdIYURLSDFmM3JV?=
 =?utf-8?B?L2ZDTS9lNktFVXV6UnFQNzdWbVgwQnJSbmw1UDl3T0t1MEdTQXRMa2ZWMGN5?=
 =?utf-8?B?TzlGMU9kQm1ocmhwcDdPVmdQeXA3Mng1L1l5bkJoRnl3T0hFR0ZQR1Uwd01D?=
 =?utf-8?B?eFZzSk1sMXZ5K1N6SFpWM1hsTFl5UEpCY1F3Q1hUZEgxeWZUT2x0cHZhZHBM?=
 =?utf-8?B?cDVsUmJIbCsyK1cwU0pvVlM4WUpwLzlVZDVlYkQ4bXBCdTFoYjdZeE5kWjdz?=
 =?utf-8?B?a2FmUFBaZzU3S3BscFN1bkxWUzFsUERmVHYraU04RkxaOTlyZEJmVVo4WXh2?=
 =?utf-8?B?bS9PSVFCUTJsSVh3cDR5cEtlWjhYclljVlFrYzIwejZGUk9BVDZiNFVUMm50?=
 =?utf-8?B?c1lDZWJmK09ybGNsYzJ2YUF6ZWxqSjl0Ni8xbXBPalVYcnRiSktTTHJSVWdq?=
 =?utf-8?B?eU5RK2hUeSs2ME1mSnVBTnRGSy9Xbk83NS9BdGpFVlU2M1RGNE9jNFlPc0ty?=
 =?utf-8?B?Z1dkNTRhSFdlcm1kdFYrL2dsWHVuWmdGODBjKzQ5aWpMb1ZRd1lKQWt3Mmcz?=
 =?utf-8?B?czBBTjRsMGV5a1pLY0Vud0g0M1ZYV2NqcTFMTldTMEk3RkQyWDBmMklsRTlM?=
 =?utf-8?B?WXhQZkVNNFROS3FLemtBajlIenRrYjB0SnBEUzA5WDRObm5meEttYlhZd0tR?=
 =?utf-8?B?cHZncWRwUTJhTFM4azVRMDlLU2dQdC9vMWtTS1RHTVpqUGsySmhpY2liS2Nn?=
 =?utf-8?B?NmNOdmQvK29PLzJsZXFwMnNLNmNXU3pPcXc5NGRzc0JEZFBzTFZ6d1hiMWdu?=
 =?utf-8?B?dElzQVhHcWt2NDd3alMxeEJUY0R4QVYwZUtiTDZJNDExR1hTMmJBSDNRYzBu?=
 =?utf-8?B?L05Qc2FIaUhJZHdTUjNVZktvU3BPZzRJNTNyY090NzU2RVM1TWY3WjgvbW5r?=
 =?utf-8?B?ZWtZWHFITVZuUVU5NDFrUWJVZXBOMVVTZ1RUdCtrK2dlQmRhdlo1cCtXV0Jv?=
 =?utf-8?B?cUhsVjJkMWd5RHd6VXpWU1loME44NlpFN0Jmcm9oWjEzeUp2MG9UME9aWndP?=
 =?utf-8?B?MEo1RWJ6SXBQZ2dtVUxzbDR5YlNDZlBxN2Q2Y2xDZlFzcGk5Q2FaT2xIcUUy?=
 =?utf-8?B?dFhnemhvSzA5MXM3YjBOV25Pa1hsMXdUVG9wcURrT0p1ay9WV3pjelYwOTZx?=
 =?utf-8?B?ckJuUG5CYWdTZVFQaXNWeVRzMXFzYzl1cGpjeUhkYkcwbU9qSHRYUnU4WlhD?=
 =?utf-8?B?OFh2UHM5OXlwcDN5cTVkOHRWdk13Y0Izcjd3QTR1MWpGQnBUUWVmRUpRdmhU?=
 =?utf-8?B?RFdmdDcwaU1EWkNJWVZQN2xTM09uVDRXUkJWUWhPYzBNc0xLMWpZaE1NY0pU?=
 =?utf-8?B?bW0rZ1R2Nkp4QzhJTnMxcEhhelhJWjg1SFN5SGgvNGozK2NBZ0FYbGhaTktu?=
 =?utf-8?B?L3JlenNTSjJNVTZHbk9RbXBVbHUxbHlqdWFBMjgwWTlYK0ZMK09HamJxTXpX?=
 =?utf-8?B?S0liZHFtcmU5ckhTakk0UTExc0xnTXNzWDhNaTJsUEJ0WjREdUM0V0J0cm12?=
 =?utf-8?B?cWNtM29LdVhjUUxueTA2c01sSEp5RnAvak9iRnVxTFBvbnJDRkdIRXJqVXZH?=
 =?utf-8?B?alRHWkNaMXpIMTc0bVpGNWlKT011ZEJIbVZuNEpsODB5L2d0RXVHTkQrU3g2?=
 =?utf-8?B?YU5GMC9hOHVBYTNMOUpDbWhIYVQ2R0xkTnRGTGNUK3BWdDZqRFRDbEJOMlA5?=
 =?utf-8?B?RG41K0U3TnU2RGltOUZJeXowZkg4UkYxTFF5WllKSkU4b0h0emRMOWQvOTBm?=
 =?utf-8?B?VlhHVEZBT2hJRXVTUUV0bnFKckp0aFVOR2VBUnJoK2ZtTHpHRGpId0dFYkV1?=
 =?utf-8?B?ZVp5Vy8xUVhwUnJ4NTNOZ0poOFdYbHFERnh5NSs2d0krUDlkWXZMYmdhdU9J?=
 =?utf-8?B?K0NXbVFqb21KYjVOVTFHeWx5ZWlJNWtDejdqV2dCV0d5RHJYM3VUR1Q2eXlF?=
 =?utf-8?B?emc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b28d8069-b075-4203-aad4-08dc4857a2c2
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2024 21:00:41.1802
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 6zaVkIs5l2kvpIod+cxD/8J4jAJDwC9exiul9DVdG+n+8Ft8AHkV/hyTg+r1+LRkGt8MPzQtqIvBCNwW1ahCZ6GBYnHGLb1Nq6MESaegEfU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB6351
X-OriginatorOrg: intel.com

Hi Tony,

On 3/19/2024 1:31 PM, Luck, Tony wrote:
>> I agree to drop patch #9.
>>
>> The core support for SNC continue to look good to me (I just had a few nitpicks).
>>
>> What remains is the user interface that continues to gather opinions [3]. These new
>> discussions were prompted by user space needing a way to determine if resctrl supports
>> SNC. This started by using the "size" file but thinking about it more user space could
>> also look at whether the number of L3 control domains are different from the number
>> of L3 monitoring domains? I am adding Maciej for his opinion (please also include him
>> in future versions of this series).
> 
> Implementing the revised user interface will make significant changes to the
> patches needed to support SNC. Working on them now.
> 
> Minor stuff is the that "size" file won't need to change (because summing across
> all SNC domains mean that all of the cache is counted in mon_data/mon_L3_xx/llc_occupancy.
> The new mon_data/mon_L3_xx/mon_NODE_xxllc_occupancy files are the ones limited
> to 1/snc_ways of L3 capacity.

If the "size" file is impacted by this change then I think there is something
wrong with the current series because "size" is intended to mirror the schemata
file that represents the size in bytes that correspond to the bits from schemata -
this monitoring change does not change this relationship.

> 
> Major stuff is that we now need both the L3-scoped domain list as well as the NODE
> scoped domain list. So no longer just changing the "mon_scope" field in the L3
> rdt_resource from one value to another.

Here may be opportunity to isolate the SNC layout from the customs. For example, the
extra layers of files only need to be added when knowing SNC is active and the
cache<->node relationships can be determined dynamically on file create and
read instead of creating new lists and data structures to try and describe it
as part of the architecture.

> 
> I'll maybe have a draft set of patches in a day or two.

I was actually hoping to get a discussion started about what folks really want
to see instead of you implementing yet another new feature that folks may
disagree with.

Reinette