Received-SPF: pass (google.com: domain of linux-kernel+bounces-195888-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <2b1b030a-1d47-4653-90ed-1caa68ad346a@intel.com>
Date: Thu, 30 May 2024 13:29:02 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v19 20/20] x86/resctrl: Update documentation with Sub-NUMA
 cluster changes
To: Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>, Maciej
 Wieczor-Retman <maciej.wieczor-retman@intel.com>, Peter Newman
	<peternewman@google.com>, James Morse <james.morse@arm.com>, Babu Moger
	<babu.moger@amd.com>, Drew Fustini <dfustini@baylibre.com>, Dave Martin
	<Dave.Martin@arm.com>
CC: <x86@kernel.org>, <linux-kernel@vger.kernel.org>,
	<patches@lists.linux.dev>
References: <20240528222006.58283-1-tony.luck@intel.com>
 <20240528222006.58283-21-tony.luck@intel.com>
Content-Language: en-US
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20240528222006.58283-21-tony.luck@intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QXUzSW8vaTErQmJycjZjcGtUc09LVFoxSy9yVlVmNlkxZUh6SW9UOFJHWmNH?=
 =?utf-8?B?dTcwaU5vajJJUDJwU1J3eWROOFJSWGtzeE54NUtaT1RxTVdCUWlLaDJCUWJ1?=
 =?utf-8?B?MXhxZjdJTmVKVHJNeTVPU0MzZWt4K1dGRGpYZUhDc09uKzlyamwwZC9iME1C?=
 =?utf-8?B?R21kNWwzc3NtWkhyTGRCajBkT2VwTGhmc2c4dFBhVnJpYmxZN01OU1VKUEp2?=
 =?utf-8?B?VzMwL1RmbUNYSTZ5RWpsdVQxU2JWdFVmRkZ0VGlhRW9SRnVaTnRnaXhvdXFv?=
 =?utf-8?B?NEoxL1Z6VHpEYWFqL2FxaWFZZWJtU3hKZVlSblM1YW50Nk5MRHdxRzM0TFlq?=
 =?utf-8?B?WXkzcGpZRkEya2tLVXl4bUZEWTJpRkJuWXhnQVU3bG00UmhKTWZMZEEwbm5k?=
 =?utf-8?B?SzNFTUZYc21sSkI1LzN2UFhzbHRPeTVBVWhJSDliOVVSRUVPVkk4dHdWVHVp?=
 =?utf-8?B?WGpFektESkRCNVFWaVUya1VURkQ5RVNFb0RpbzcxK25ER2dZOWZIcU9WMnRz?=
 =?utf-8?B?Tm1GbzNiM0s4T1M1K1Z3djlBYzFFVTlTb0ZxdHByeVJnU2wvQ0JWd1JsWTZM?=
 =?utf-8?B?WFVLTFk4N015elZzb1VrVXo4VlJ3WlNManZpWFFRemRQZ0tBZ2JuWWpGVjNO?=
 =?utf-8?B?TVhJb2JkaFB2Y1ZsaHlCeXdjQ0N3dWFEdy9hc25nN3VMMjlTUDloVmlJL0pE?=
 =?utf-8?B?R3dhMFBSanBQRERsYVBlbk1hSk93TjhmMGwvQjBlSG1OK1hPOFNIMGpzYnh4?=
 =?utf-8?B?Y2M1dDVKSkV1Njh5K0pjSlB0Ynk2YS9yMlpOM2F6L2J3SHFOT2pETEsyNVZI?=
 =?utf-8?B?M2YxQ1JTczc4eE1oMzVvdkhkOEprYjJUV0hJM1p4ZWN1a25tM2N3UDlqWjk5?=
 =?utf-8?B?eHFsT3JMRU1nd0hEWXp1ZTJ1bUEreHBNZmNxSUZDNmhIdnYxbytyNWtYYnQz?=
 =?utf-8?B?OU4vVUJFYjJFL1lvcXpFU1lMc1BvcDYwaFpyR1hRWm9PNzVGTlV5ZWJGQTUz?=
 =?utf-8?B?eUJHaElUTTducXFHS3BEaXVibW8vYk9iQVlBVEF3eVNmOGRINUI3Z0tubHRD?=
 =?utf-8?B?ZXBTRkIwNVorWVIya2ZnT1NKc1FhV3AvTC9GNVZabWMyYW1YdkpyRDgyTjVR?=
 =?utf-8?B?WXA4SWt4MW5pNVI2b1FhcnBmUlBCeVpnWXlDb25iTEZXRmljZjhnK2Y5a1hx?=
 =?utf-8?B?K1c4N1Y0czVEUVVzQ0ZnajM5cGtGMnhiV0xEeDc1YmFaYzJhOUY4OUlHUmNC?=
 =?utf-8?B?ODJ0clJVZXMvWlRMMHRXVGVtQnhWL2lVVTdOUXZ2OHVLcERHMTk1Z1lYN3Fq?=
 =?utf-8?B?Szd1YjJsOWtzbitURlo0aDVCOG5GUGNhdldTUno4a2VhVFpSN1M3UnkxT3dU?=
 =?utf-8?B?Y0wxdisvTWxIS1oxY08vNThpYUVZYXFYQUJ0QWtWNkNON1VieEpvRkZBT3Az?=
 =?utf-8?B?UHk0VGZhbjlUNUZSaWhHZDd5UWV2WmFWTnRpL3J1d1dQamRXQVhUbkM4YlFr?=
 =?utf-8?B?ZEpLeERzcDRHcytYM0d4OWVVQU84c1ZjMEFwbkprN0c0VDNFRENQdW0yUksx?=
 =?utf-8?B?UlZxb0hZTzJYb0xMeHNHb1JtQUVyMllxVXg1WjZ0OXJZWUJxOURIVnBoK1Ir?=
 =?utf-8?B?dm00K0E1ajZSaDN2d29LcS9tMkhydHBtRlYyRUFlTzFRMXgzL3hZb1l5YWVL?=
 =?utf-8?B?MGhjakJTL2M5QVVzaldvbWlPQXoyUnJqcXBtOEYyNkJQS0xFZnlqKzZKMGta?=
 =?utf-8?B?RnFlY0dRbExzZXUyOTRMaWlNS2MxVkhaako2Tmd1cHVQTW5La3FacDgrQXZj?=
 =?utf-8?B?Y3JabzlEZWw3bnFJbFVqTXIxdEsxUE56NFJGRjlVNFBrUHlUVC8zSEY3S3Rt?=
 =?utf-8?B?WGRiZ2QzSjk1VS9uV2VUNlhNcER2dllBVUMwczhIckhiK2xqOTdWTGhKVXNM?=
 =?utf-8?B?MXMxNlJzcGtsTk9XOEZXRERPM2g5amhvSkROaW0zWVliZWJDZmJvYXZyd2Vj?=
 =?utf-8?B?aWFQU0dzVWpHZVkvUnliQ2FxY2U5V1pHS0lvNzZnQ3N3cEM5aG5lUVRRUk9y?=
 =?utf-8?B?ODI4ejhyMXBnNVpKNGxNVFJTY1dSWXRGc2EvbksxOFVkYTg4bWdxeDJOTGFZ?=
 =?utf-8?B?d2dLc3F5SEpKZlJub25zTWNKRlZRZ1h2Y2pxY3hZSHNyTmo3M05wdE9BT3N1?=
 =?utf-8?B?NWc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 48f48860-06b3-4c61-5a52-08dc80e7261f
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2024 20:29:04.6214
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: cW0PpQ0B4FUI5KEol7uX660RbGyktzJ8S52OtVfCWlfX/wyxJBK/UFV2TrYfSCuwPV4Nh35pQF6HSvS6S67KPZ9mC5V5Hyfrvc1Qd4HzSiU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7317
X-OriginatorOrg: intel.com


On 5/28/24 3:20 PM, Tony Luck wrote:
> With Sub-NUMA Cluster mode enabled the scope of monitoring resources is
> per-NODE instead of per-L3 cache. Suffixes of directories with "L3" in
> their name refer to Sub-NUMA nodes instead of L3 cache ids.

(insert feedback from v18)

> 
> Users should be aware that SNC mode also affects the amount of L3 cache
> available for allocation within each SNC node.
> 
> Signed-off-by: Tony Luck <tony.luck@intel.com>
> ---
>   Documentation/arch/x86/resctrl.rst | 17 +++++++++++++++++
>   1 file changed, 17 insertions(+)
> 
> diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
> index 627e23869bca..401f6bfb4a3c 100644
> --- a/Documentation/arch/x86/resctrl.rst
> +++ b/Documentation/arch/x86/resctrl.rst
> @@ -375,6 +375,10 @@ When monitoring is enabled all MON groups will also contain:
>   	all tasks in the group. In CTRL_MON groups these files provide
>   	the sum for all tasks in the CTRL_MON group and all tasks in
>   	MON groups. Please see example section for more details on usage.
> +	On systems with Sub-NUMA (SNC) cluster enabled there are extra

(insert feedback from v18)

> +	directories for each node (located within the "mon_L3_XX" directory
> +	for the L3 cache they occupy). These are named "mon_sub_L3_YY"
> +	where "YY" is the node number.
>   
>   "mon_hw_id":
>   	Available only with debug option. The identifier used by hardware
> @@ -484,6 +488,19 @@ if non-contiguous 1s value is supported. On a system with a 20-bit mask
>   each bit represents 5% of the capacity of the cache. You could partition
>   the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000.
>   
> +Notes on Sub-NUMA Cluster mode
> +==============================
> +When SNC mode is enabled, Linux may load balance tasks between Sub-NUMA
> +nodes much more readily than between regular NUMA nodes since the CPUs
> +on Sub-NUMA nodes share the same L3 cache and the system may report
> +the NUMA distance between Sub-NUMA nodes with a lower value than used
> +for regular NUMA nodes.
> +The top-level monitoring files in each "mon_L3_XX" directory provide
> +the sum of data across all SNC nodes sharing an L3 cache instance.
> +Users who bind tasks to the CPUs of a specific Sub-NUMA node can read
> +the "llc_occupancy", "mbm_total_bytes", and "mbm_local_bytes" in the
> +"mon_sub_L3_YY" directories to get node local data.
> +
>   Memory bandwidth Allocation and monitoring
>   ==========================================
>