Received-SPF: pass (google.com: domain of linux-kernel+bounces-106561-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <47c28b3d-3b90-4b33-ba71-1e359d43238e@intel.com>
Date: Mon, 18 Mar 2024 12:15:57 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 4/4] selftests/resctrl: Adjust SNC support messages
Content-Language: en-US
To: James Morse <james.morse@arm.com>, Tony Luck <tony.luck@intel.com>
CC: "Wieczor-Retman, Maciej" <maciej.wieczor-retman@intel.com>, "Yu, Fenghua"
	<fenghua.yu@intel.com>, Shuah Khan <shuah@kernel.org>,
	"ilpo.jarvinen@linux.intel.com" <ilpo.jarvinen@linux.intel.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>
References: <SJ1PR11MB608310C72D7189C139EA6302FC212@SJ1PR11MB6083.namprd11.prod.outlook.com>
 <o6va7b7rc3q46olsbscav7pla4hxot2g6xhctflhmf64pj5hpx@56vtbg3yyquy>
 <SJ1PR11MB60830E546B3D575B01D37104FC202@SJ1PR11MB6083.namprd11.prod.outlook.com>
 <159474e6-ef11-4769-a182-86483efcf2a6@intel.com>
 <SJ1PR11MB60832DAD58E864F99A16FCC4FC202@SJ1PR11MB6083.namprd11.prod.outlook.com>
 <0393c4ce-7e41-4dcc-940a-a6bea9437970@intel.com>
 <SJ1PR11MB6083AACB10645E41DD3F9639FC202@SJ1PR11MB6083.namprd11.prod.outlook.com>
 <55a55960-8bb1-4ce2-a2c7-68e167da8bcc@intel.com>
 <ZepK4mtoV_J8-UbE@agluck-desk3>
 <eacdc287-24bd-4137-85c8-df055cfd78b1@arm.com>
 <ZetcM9GO2PH6SC0j@agluck-desk3>
 <88430722-67b3-4f7d-8db2-95ee52b6f0b0@arm.com>
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <88430722-67b3-4f7d-8db2-95ee52b6f0b0@arm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aS9tZWZOanpZeXBLUkVDa1FXMVkyZHFvR1VYZ3dhcUJpUU96dTcwUUV0RXgr?=
 =?utf-8?B?R1JjejlRSVRxTzdjWG1zMlROZjlpT0U5RmZMZnFQWHptSjkvbW04cm13Ung2?=
 =?utf-8?B?TXBMRHMwZEpyZEp5OGJoQ3RsUnBLTGFwUVNwTmw3bGRmR055QUpDTWJkQWov?=
 =?utf-8?B?d1JORXJxTW96SjFnZ29nOUFmWFo3Y1JMQzYrNEtvb2k2eHkzN0hKTlAreWF5?=
 =?utf-8?B?Qlk0Zk8ydEZnYXpybXB0QWJnS3NpSUF5VXBBbGIzZTRDc3J6ZkhDOEJvSFZC?=
 =?utf-8?B?UlllZTB1U01DbXJLWVFvVWFFS1RrSG9rNngxa09uRFI0NDRuUTRpc3hpS210?=
 =?utf-8?B?UnJra29VdlJrSVVIbXMvUEttMjRPMzdsaUlJeTZWVFU3dzFqcWpLbkkrZXFp?=
 =?utf-8?B?REl5bDljcCtBSjVGWk1sRVZ4Wlc3bStsL3VXejN4bHBGUE1WOXZGTVVINFBr?=
 =?utf-8?B?N01ZRHRxUzFxaEc4Sk4wQjNza2VKKzN5S1RxMjVKYjRHcVovejR5TXJ5eHRQ?=
 =?utf-8?B?S3o2OSs2Y3lRWTNrT0NmZUdUQi8vK3BVVmJOQ2tCOWJCWUM2alFPM0pnei83?=
 =?utf-8?B?K0lBUytHbVRKbVV5OHh4S2t5MkgzeHVRM1B2cnl4eUNidDU2RWErNDVtK0RG?=
 =?utf-8?B?V0JTVGhva1pxMVF4T3dJZG1rMXJRSUN0QWZvZ0tlNXZQcUxEMVk4MG9oa3lO?=
 =?utf-8?B?bVF5VnF0NU9GKzVReVFBcE8vQUtuVUl2Z1FKY1gybmx5aFZ3anpZL05nR0Y5?=
 =?utf-8?B?anNRc254T3k4dWd0SFlnbnNZdUJGMjlxYmZJaGVnMW96ZWNKck9qY0FFdCti?=
 =?utf-8?B?K0tXSVBXUmdqVnZqSXJTRkUwQzBFSFpDU0lrOFNWZ2xEY3BNTE1hNlVDTVpo?=
 =?utf-8?B?MmlvclU5SG8wTUhuUXJIOWEybzExK3RwNkJ4amFlcFZkWHJydWRMVU12bDN5?=
 =?utf-8?B?aVdMOUZ2MEd0bS85ZHg3ZkFZSzJGNzB6b0p2ZGhpNmVqc2dNYjNjWFJiWlpJ?=
 =?utf-8?B?Vm1sYnVXYWZablZtMGJrY1Q5UXdUL1dWTDBDM0dWanlYUjRiY3ZUTG9WNHh4?=
 =?utf-8?B?emxYb2o0Rm5OelFrRmozdm5tbVNOckZxTk5Bb2tLMjlvNmJkTkJFMkJ1REFr?=
 =?utf-8?B?ampxMThzSm90Qmsvcm5CdUtpM0RySDhnZ3JERDVOaEZaMkRWOWdwTGl5dlRs?=
 =?utf-8?B?SWFFOEFnU0l3NHhrNlN4NldmbXlqamVGQWVKY0RRVG83Qit4djNxbUhGam1F?=
 =?utf-8?B?M1I0YXU0UG9OZVFxemY1N3R4eWllLzdGWFptcDJSNGVCTEZwazRrUERpZmZD?=
 =?utf-8?B?QmJ5amRwWmJzZ0hQeDVGNUMzSURUSks5NDRVdUs4dTAvR3dyS1Q1U0RJTTNJ?=
 =?utf-8?B?ckVvSTN6bHBaa0xzR2FMTUpCZnpVMEZQY28vb0tONTJ4Wi9EQjk4UXI0VkF6?=
 =?utf-8?B?cWNyWWNmNUJDWlZ3Wno2UHU1WjNrbEZPOUhQeCtEU2lBa3NPYzM4VnZRV2tU?=
 =?utf-8?B?QW0wYVJMZW9EcHJuS3hGNVlFeTV1c01lTGl6cERvaVFUZ2lrVSs3TXdiK1Qy?=
 =?utf-8?B?K0hrZ1JDWk1zd0ZweUxlR3hvMWJJS1ptU0JOYWhXTGtJZE1oUllwMkVQTzBs?=
 =?utf-8?B?anFQalVZZlJZNHpMZ3dGUzAvRTc1ekhsbE9VdVpSYjlVSXg2bkkxdWF3a3lW?=
 =?utf-8?B?NjhOT3ZDUkhMNkhYTXFlOGhiRk5JaVJ3eWVoZzN3TDB4dWxHNDArc0lJZDRy?=
 =?utf-8?B?Q0RGUloyQjlreG12SDFUb2tvazkzNk5oanpjU1lKOEQ4Wk9HRTZ4QkZ0cDcy?=
 =?utf-8?B?K0RvcVc5dFk2TmUrZmFnMnJOeFBjbktPZU1CY0VZTDI5MktGMGlGekRkbUNo?=
 =?utf-8?B?MCtNd3J1YzdFUVdKblZzOElLemhEN0N1MllMazA3dzMvd0xuMnJyRlpJQWxT?=
 =?utf-8?B?QmhNNUlCbjhsRVVuSVVHcWlVdCt0Nng0V3g1eFlWYnBibjRpbnhiclBJRW03?=
 =?utf-8?B?c1l1MERhM0VrMTZ3ZjFMRjluUVBvQ2EvMTNVVm8wREpnZG9jNEdVQ2lldDNW?=
 =?utf-8?B?VnVKRG5TTmhPMmxmcWR6aHJQdThPdGMrWG9DcU1jSURmNTRRWmhabEpGRlBw?=
 =?utf-8?B?WGo2dWZ5a2xSNVJ3Q3BrWThkOGl3UVp5NzJIS1prUWhjci9saGdPNXZIQ3RK?=
 =?utf-8?B?aHc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d5cc809b-96dc-4902-19a7-08dc477fd83b
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Mar 2024 19:15:59.5712
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: OYv7ZGN1oq2siQoteVZkDP6/7EV1uAKf958XL5id7xY2Y7FKpSVHnazRc4QQ86/k0wz4gAkXMC4AnHy193iAYDrpXCEfKmoxNVUDPYJ75+0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB7007
X-OriginatorOrg: intel.com


On 3/15/2024 11:02 AM, James Morse wrote:
> On 08/03/2024 18:42, Tony Luck wrote:
>> On Fri, Mar 08, 2024 at 06:06:45PM +0000, James Morse wrote:
>>> Hi guys,
>>>
>>> On 07/03/2024 23:16, Tony Luck wrote:
>>>> On Thu, Mar 07, 2024 at 02:39:08PM -0800, Reinette Chatre wrote:
>>>>> Thank you for the example. I find that significantly easier to
>>>>> understand than a single number in a generic "nodes_per_l3_cache".
>>>>> Especially with potential confusion surrounding inconsistent "nodes"
>>>>> between allocation and monitoring. 
>>>>>
>>>>> How about domain_cpu_list and domain_cpu_map ?
>>>
>>>> Like this (my test system doesn't have SNC, so all domains are the same):
>>>>
>>>> $ cd /sys/fs/resctrl/info/
>>>> $ grep . */domain*
>>>> L3/domain_cpu_list:0: 0-35,72-107
>>>> L3/domain_cpu_list:1: 36-71,108-143
>>>> L3/domain_cpu_map:0: 0000,00000fff,ffffff00,0000000f,ffffffff
>>>> L3/domain_cpu_map:1: ffff,fffff000,000000ff,fffffff0,00000000
>>>> L3_MON/domain_cpu_list:0: 0-35,72-107
>>>> L3_MON/domain_cpu_list:1: 36-71,108-143
>>>> L3_MON/domain_cpu_map:0: 0000,00000fff,ffffff00,0000000f,ffffffff
>>>> L3_MON/domain_cpu_map:1: ffff,fffff000,000000ff,fffffff0,00000000
>>>> MB/domain_cpu_list:0: 0-35,72-107
>>>> MB/domain_cpu_list:1: 36-71,108-143
>>>> MB/domain_cpu_map:0: 0000,00000fff,ffffff00,0000000f,ffffffff
>>>> MB/domain_cpu_map:1: ffff,fffff000,000000ff,fffffff0,00000000
>>>
>>> This duplicates the information in /sys/devices/system/cpu/cpuX/cache/indexY ... is this
>>> really because that information is, er, wrong on SNC systems. Is it possible to fix that?
>>
>> On an SNC system the resctrl domain for L3_MON becomes the SNC node
>> instead of the L3 cache instance. With 2, 3, or 4 SNC nodes per L3.
>>
>> Even without the SNC issue this duplication may be a useful
>> convienience. On Intel to get from a resctrl domain is a multi-step
>> process to first find which of the indexY directories has level=3
>> and then look for the "id" that matches the domain.
>>
>>> >From Tony's earlier description of how SNC changes things, the MB controls remain
>>> per-socket. To me it feels less invasive to fix the definition of L3 on these platforms to
>>> describe how it behaves (assuming that is possible), and define a new 'MB' that is NUMA
>>> scoped.
>>> This direction of redefining L3 means /sys/fs/resctrl and /sys/devices have different
>>> views of 'the' cache hierarchy.
>>
>> I almost went partly in that direction when I started this epic voyage.
>> The "almost" part was to change the names of the monitoring directories
>> under mon_data from (legacy non-SNC system):
>>
>> $ ls -l mon_data
>> total 0
>> dr-xr-xr-x. 2 root root 0 Mar  8 10:31 mon_L3_00
>> dr-xr-xr-x. 2 root root 0 Mar  8 10:31 mon_L3_01
>>
>> to (2 socket, SNC=2 system):
>>
>> $ ls -l mon_data
>> total 0
>> dr-xr-xr-x. 2 root root 0 Mar  8 10:31 mon_NODE_00
>> dr-xr-xr-x. 2 root root 0 Mar  8 10:31 mon_NODE_01
>> dr-xr-xr-x. 2 root root 0 Mar  8 10:31 mon_NODE_02
>> dr-xr-xr-x. 2 root root 0 Mar  8 10:31 mon_NODE_03
> 
> This would be useful for MPAM. I've seen a couple of MPAM systems that have per-NUMA MPAM
> controls on the 'L3', but describe it as a single global L3. The MPAM driver currently
> hides this by summing the NUMA node counters and reporting it as the global L3's value.
> 
> 
>> While that is in some ways a more accurate view, it breaks a lot of
>> legacy monitoring applications that expect the "L3" names.
> 
> True - but the behaviour is different from a non SNC system, if this software can read the
> file - but goes wrong because the contents of the file represent something different, its
> still broken.

This is a good point. There is also /sys/fs/resctrl/info/L3_MON to consider and trying to think
what to do about that makes me go in circles about when user space may expect resctrl to indicate
the resource and when user space may expect resctrl to indicate the scope. For example, 
/sys/fs/resctrl/mon_data/mon_L3_00 contains files with data that monitor the
"L3" _resource_, no? If we change that to /sys/fs/resctrl/mon_data/mon_NODE_00 then it
switches the meaning of the middle term to be "scope" while it still contains the monitoring
data of the "L3" resource. So does that mean user space would need to rely on
/sys/fs/resctrl/info/L3_MON to obtain the information about which monitoring files 
(/sys/fs/resctrl/info/L3_MON/mon_features) are related to the particular resource and then
match those filenames with the filenames in /sys/fs/resctrl/mon_data/mon_NODE_00 to know
which resource it applies to and learn from the directory name what scope measurement is at?

Reinette