Received-SPF: pass (google.com: domain of linux-kernel+bounces-180804-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Thu, 16 May 2024 10:57:21 +0200
From: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
To: Tony Luck <tony.luck@intel.com>
CC: Fenghua Yu <fenghua.yu@intel.com>, Reinette Chatre
	<reinette.chatre@intel.com>, Peter Newman <peternewman@google.com>, "James
 Morse" <james.morse@arm.com>, Babu Moger <babu.moger@amd.com>, Drew Fustini
	<dfustini@baylibre.com>, Dave Martin <Dave.Martin@arm.com>, <x86@kernel.org>,
	<linux-kernel@vger.kernel.org>, <patches@lists.linux.dev>
Subject: Re: [PATCH v18 17/17] x86/resctrl: Update documentation with
 Sub-NUMA cluster changes
Message-ID: <az7n7nj6bl5ixyb2xxdoxtjaye4kbhm2bw6sdz63h2btvo3n7a@bo4hlxxtu4nf>
References: <20240515222326.74166-1-tony.luck@intel.com>
 <20240515222326.74166-18-tony.luck@intel.com>
Content-Type: text/plain; charset="iso-8859-1"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20240515222326.74166-18-tony.luck@intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?kwzyre1YDd2m45X39SPLWqevxrKAIBv8J2JMp7Q7JAc+9DCyLz0Mh7fzr8?=
 =?iso-8859-1?Q?HBVU724PC2qBH/hKBo91VCAvLpTwLsLiB5PbFQSZ0ZwMg9uWRpSQx4oIeO?=
 =?iso-8859-1?Q?5SrcnhoMelcI/Da/AkccFHvVKpdAOh2WVdYF0bfGfTcs3Vbxl11paUNaO2?=
 =?iso-8859-1?Q?QZzB32vYheo2RDtmNzttl7FDqF8kjFJs/VEdZ1RmAfrQ+advcAB/nmBgyJ?=
 =?iso-8859-1?Q?3odhSF4lhAGlKOGwxeSqLoAeFgnBhmuS1J7iLGf/0PGnH0iSWwxC3FFpjs?=
 =?iso-8859-1?Q?nDYJkt/3X3G5dtgNxWINyNXUEKzKtvrtQZ5MJusxQAS14UDZMtuQNM0+J5?=
 =?iso-8859-1?Q?pR1pkoUbHgKhv9t7ddEmcyQixMWj8V6bcmedjLcyzi5AMGltqD0tvs+KP/?=
 =?iso-8859-1?Q?gBQgYOxPpCkcKQ39IMbGEq9y4ihMIJ2OeDAe62SaJCjlNudacTu7rSpQZ9?=
 =?iso-8859-1?Q?CrOeiaJXoJwzC8uvNPTP36XHGN5QUwMBAKoAhJNvZlTtzbMj6WGyQYBE9U?=
 =?iso-8859-1?Q?zfenQxB6n94jcmAjhMTZ/Xm4dVhEwQOCo+HAuAJ3huUT1WFrCFzX1fV03v?=
 =?iso-8859-1?Q?RdxUasycWzUtXtj1hDqSo7gc2RsdrKGQpP3oHxx6uwh6I3e1TQ+DzL05pF?=
 =?iso-8859-1?Q?5oSMjK/vOvSfX7jjIOOFAVlc5fwgjTL5knc/GTR8pEILAi4BKGGNUlS7yr?=
 =?iso-8859-1?Q?RSjc0TSC5HCpAv/X0sBkwHdcaYfxKKJ+AstgZYKXNEVR5IwqwClBscPWUH?=
 =?iso-8859-1?Q?RJPiVr73a+aDvkTT9a87Eloi2ml3UFeOCAKVcMZvzpHxFfMNJfdWTJ5H7r?=
 =?iso-8859-1?Q?UqxgIiIuqQA/SV06L5RLTZez8CWbG2xvOGZ0xL+gaQOmJHtT58kP61Tm5X?=
 =?iso-8859-1?Q?Ob3dRNlZi35ClNHenOS6vs03uIuT0sLu7aIb9wDxhWOvrRTvMdq0C3xEll?=
 =?iso-8859-1?Q?YTxvvAntpVLtjaq8J7WHD57YuBz50IGqIP+fyfKeYCYvNfDnLEDU30nP1E?=
 =?iso-8859-1?Q?TaeD+i5kC2iuOiLwgSUIeE44uh4yoov7s8+ur6CttPElI2tMg0Y9TLi3hR?=
 =?iso-8859-1?Q?glBQ21x3Sdmv/gTlxU+u7NgaoTnW5rtl+7AOC0b8/n7cm512dtAcmLQzDi?=
 =?iso-8859-1?Q?IlKdVqLZQ1gOQMT2yCheLUc8BFEEjsDDZ9DZ0MQ6tjWReH8c+9+LJsMPUd?=
 =?iso-8859-1?Q?iwjnsNTgWzz4wV/Q2hjshF1eflzdxKgQ3IsYlK4LujEO+XBOsAoF4aKZEb?=
 =?iso-8859-1?Q?QlPWEJAS+8cjWF81K+cauGs/7cfg9tFtILLPkB/iAEB+0dfUSf6x+lAPLp?=
 =?iso-8859-1?Q?3da/rbTiLfPgsIUkW/eZ3PDSxom5qu2bl24qp37QYr/yctdmj0EHiSDDF6?=
 =?iso-8859-1?Q?chTwFEnvhPJIKqbfaTDSpKlOGU6uSLoh6L2QPHLL6v8hba27hlc1TJXHBd?=
 =?iso-8859-1?Q?FQOmZ3pP0F2sioDw9C5IosNphk4vf65pmQfafXDaGMzcdOJZebk+1hPAW/?=
 =?iso-8859-1?Q?vKqzUfM4Ehduw87iRFMpgTstafKTi+KrnjuU9uRlmHKmy0Yrj9AKMl+/Qr?=
 =?iso-8859-1?Q?aNN4zjXDcHVNhpUdVxseaeomD0fwQI1Q7anko/7+R3ze+DshzUZHqszliP?=
 =?iso-8859-1?Q?dQh4Bvky6LVXbyIcVPOfUWCS6rhqRxCbWsN9EUTumBtRuvSPp5G0L+fcGs?=
 =?iso-8859-1?Q?MJhNFQV05BF5tuqguBo=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: cc18a993-e3aa-4a4b-c30f-08dc75863aa9
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6231.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 May 2024 08:57:35.1893
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 8EskTCcjBaueGqUVJ7WF0k69zCIdKJ4BUs3ofnUoSjsjL6xEEiEBRf3NvSMYI5L4KjxZ4dJ38ibR+lv170a6mgXRuorq9oJ9mSpmQ9TqoLM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB8668
X-OriginatorOrg: intel.com

On 2024-05-15 at 15:23:25 -0700, Tony Luck wrote:
>*** This patch needs updating for new files for monitoring ***

Is this note here by accident? New files seem to be mentioned in the patch.

>
>With Sub-NUMA Cluster mode enabled the scope of monitoring resources is
>per-NODE instead of per-L3 cache. Suffixes of directories with "L3" in
>their name refer to Sub-NUMA nodes instead of L3 cache ids.
>
>Users should be aware that SNC mode also affects the amount of L3 cache
>available for allocation within each SNC node.
>
>Signed-off-by: Tony Luck <tony.luck@intel.com>
>---
> Documentation/arch/x86/resctrl.rst | 17 +++++++++++++++++
> 1 file changed, 17 insertions(+)
>
>diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
>index 627e23869bca..401f6bfb4a3c 100644
>--- a/Documentation/arch/x86/resctrl.rst
>+++ b/Documentation/arch/x86/resctrl.rst
>@@ -375,6 +375,10 @@ When monitoring is enabled all MON groups will also contain:
> 	all tasks in the group. In CTRL_MON groups these files provide
> 	the sum for all tasks in the CTRL_MON group and all tasks in
> 	MON groups. Please see example section for more details on usage.
>+	On systems with Sub-NUMA (SNC) cluster enabled there are extra
>+	directories for each node (located within the "mon_L3_XX" directory
>+	for the L3 cache they occupy). These are named "mon_sub_L3_YY"
>+	where "YY" is the node number.
> 
> "mon_hw_id":
> 	Available only with debug option. The identifier used by hardware
>@@ -484,6 +488,19 @@ if non-contiguous 1s value is supported. On a system with a 20-bit mask
> each bit represents 5% of the capacity of the cache. You could partition
> the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000.
> 
>+Notes on Sub-NUMA Cluster mode
>+==============================
>+When SNC mode is enabled, Linux may load balance tasks between Sub-NUMA
>+nodes much more readily than between regular NUMA nodes since the CPUs
>+on Sub-NUMA nodes share the same L3 cache and the system may report
>+the NUMA distance between Sub-NUMA nodes with a lower value than used
>+for regular NUMA nodes.
>+The top-level monitoring files in each "mon_L3_XX" directory provide
>+the sum of data across all SNC nodes sharing an L3 cache instance.
>+Users who bind tasks to the CPUs of a specific Sub-NUMA node can read
>+the "llc_occupancy", "mbm_total_bytes", and "mbm_local_bytes" in the
>+"mon_sub_L3_YY" directories to get node local data.
>+
> Memory bandwidth Allocation and monitoring
> ==========================================
> 
>-- 
>2.44.0
>

-- 
Kind regards
Maciej Wiecz?r-Retman