Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36;
Message-ID: <7b66283e-5801-ee37-aa76-af0c04240a20@amd.com>
Date:   Mon, 4 Dec 2023 17:04:30 -0600
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
From:   "Moger, Babu" <bmoger@amd.com>
Subject: Re: [PATCH v13 8/8] x86/resctrl: Update documentation with Sub-NUMA
 cluster changes
Reply-To: babu.moger@amd.com
To:     Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>,
        Reinette Chatre <reinette.chatre@intel.com>,
        Peter Newman <peternewman@google.com>,
        Jonathan Corbet <corbet@lwn.net>,
        Shuah Khan <skhan@linuxfoundation.org>, x86@kernel.org
Cc:     Shaopeng Tan <tan.shaopeng@fujitsu.com>,
        James Morse <james.morse@arm.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Babu Moger <babu.moger@amd.com>,
        Randy Dunlap <rdunlap@infradead.org>,
        linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org,
        patches@lists.linux.dev, Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
References: <20231130003418.89964-1-tony.luck@intel.com>
 <20231204185357.120501-1-tony.luck@intel.com>
 <20231204185357.120501-9-tony.luck@intel.com>
Content-Language: en-US
In-Reply-To: <20231204185357.120501-9-tony.luck@intel.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UnBZYWl0RWYrNXRVNTIrVklWY0NVSmpmSTEzYktlQzRvUlh3OU5rM2lBUFNp?=
 =?utf-8?B?RXJ0WE50YTdkeFNTSTU4ZGNxcExtbEtlTHk1WDMxZlFoWnhWWGFIbmEvWVJs?=
 =?utf-8?B?QmoyMWZFOEhVRmNXWnBVSEgwa3NXUVVTcmVTN0tRclZ5ZElLUWV0TjhEWStD?=
 =?utf-8?B?dVRYYjhzRURwd2FyZ1UzL3pra3ZDT25yaHliS2RtUm9uMUpuOU9sUDY0QWpV?=
 =?utf-8?B?K0wvcjBuMzFEQkE4cXcxc0Z6YW1sM2N3V0tLVy9rYXhYMW9UZnppd1lqeEp1?=
 =?utf-8?B?ZmxWSEx0R2Njdk9hNmw2TnpQamtPOWFSNXFTbkgyMXZ4Yk85MzIycTlZcHQ4?=
 =?utf-8?B?QVJWVWFFRFB1dXp1VWRsMkNwNjk3OFRLMFVvTHhjMzJUM2hxWWkvTEcwSkpS?=
 =?utf-8?B?bzdQUE80UnNqbXVYdm5EaUNrRGk3SHd0RzRuSEtkRHhrUmxwblVyTk9ndDVv?=
 =?utf-8?B?M3Nxd3Z3bC9LQjNVMElWb2hKL2tMaVBSSElmWkFFRmFPejhETGRIV1MrOHU4?=
 =?utf-8?B?OEM0NDhNeDhseStGSXBDYVQ0c0pId1pvZkNobmFjUHRSTFpxTjJjNE5Nc2pO?=
 =?utf-8?B?V3BNWHdKeEt3UXpuQThTTVEzTnJqbmI3VTIxK1FKZ2FIZlJaUVNJM2czSmVY?=
 =?utf-8?B?VDZBcXR4OXpsVW04Q2NIRWgvbkhOT1B2UGpWWTZwS2NZMUVuNmVDR1RZNFRn?=
 =?utf-8?B?ZkxUbjhGSGk4Szk3ZWhwQ0ZaS2E1SFJDcTFUN2UvU01iNjBCciszcG5Td3RU?=
 =?utf-8?B?RkxXeWJ6ZWJ1ZTNkeUp6MkRDY1plTTRaa2FoR3lOMWkvaGZkSnBRU29kOHRT?=
 =?utf-8?B?cDdWOXBZcWZmNTlRU25iTW5zRzJRUDBnSFczREtDb0Qvd2JLRG12a3RHUnBz?=
 =?utf-8?B?S3pIOHNTWG9wOTM4aEp4RExCSGZPUDYvNlVpL2dmbFlWelppSlg3ZWFpaXdI?=
 =?utf-8?B?NEl5NEp0ekplY3ZsYkRVZzBLMk5vZTlMNndPYzVkRlErY0hVSVhRQWhqMTI5?=
 =?utf-8?B?ejByZDM3dklML2hLdzFtNE9QWTZIWjJjUmwyT29Zcm0ybHhDQTdGczVXM2JU?=
 =?utf-8?B?Vkk3VzM1VnJNMDY1V2ZLVlM1REx6LytvbnR0U0ZYaDBJQzVMRU9JK3F3cHd6?=
 =?utf-8?B?NkVmNXBRQnE1aWVhOWJCaDF1ZkpmL3oyV1J1M25vc3dZZFM3WXJhV3J2YjBq?=
 =?utf-8?B?c25EWmpSNXNjRUR3OGcvYkU1ZEhwSWRlZUtselpXdVhTajBxbkdHVjg5aHkz?=
 =?utf-8?B?d09zSXFnVlVqM2E3SVB0dVdTNGplTXpLMk1DcUh6VTJ5TitMZElZSzZEMHNX?=
 =?utf-8?B?YjRpdjZpbkVybTRuWndLcFRPMDd2Z3lqR1k1ZHBjRXFvbHJsYm83VTV1UkdE?=
 =?utf-8?B?ajRteHFtdDI3SXA0S2VkWjZBVksrazdCUlZ5a1k3MjdFYjBQNXROd0hPQ2dl?=
 =?utf-8?B?TnZhYTJtVkQrOWxnVkFZaHl0RTNFTGZGZ29PT0ZyMkZSZ2daZy9EV3VGcFdw?=
 =?utf-8?B?YXNJRHlHMDgvaWZINll2d2E3dldhQzJEV2o3SjI4QnFxazB6cytaa25ta09B?=
 =?utf-8?B?MlRmNWtMMkhHM3RkQW81dnU2YlI2Mk1YZ1ozcjVkOWhmV003RnhvSElYQ0Er?=
 =?utf-8?B?dURwWDhwKzl5Sm5VUHZLNlpoYTNqOS94L2lIRWhjUGlreHA0cTZ5RTgwRnVY?=
 =?utf-8?B?RjB1NmpNL0R5dkI1MU1WUStOc2VZY3NPSERVNHZlSTd6cTBpaUMrdjNQY25K?=
 =?utf-8?B?Unc1bHdyb1pYa1lqcm9PbVFZaFZtNnVIVmtOVGZ4NlVQWlc4UkJFNnRkT0tj?=
 =?utf-8?B?cXpCQkhQL2xOZTNCZnNZVDl2RHU2WHZwQjdOM011K2pxWThUUjVIMks3SEpP?=
 =?utf-8?B?b1MwUHVNU2NrcFZTRHdnV3JHbDByeCtyLzhPMUkySFVEak1JS252dThXS0I1?=
 =?utf-8?B?MXBYR3k3VlBub0JHaGhWdTVhYTFWMzA0MmRxbjJIOUN4clRDcm9kSlA4aUpY?=
 =?utf-8?B?WlNtVWJ2eXBSeXc5aDBCdDVQQUlxOWcyNE4ydGhNMDFMY3ZMWmFkRXUrM040?=
 =?utf-8?B?RzJZRXhiaElPVEExV1FzTUJXZTBnclBkR3F2THZUMHZrTzR2RGlxb2t3bEVw?=
 =?utf-8?Q?MFyNCh0GbikEEqcEBj5/P4Eh3?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 08139f6d-245e-4837-2156-08dbf51d6198
X-MS-Exchange-CrossTenant-AuthSource: MW3PR12MB4553.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Dec 2023 23:04:34.4384
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: PfkneCru5ur5BacAod7f/Y7CMjB6IG2EM1WjE7IEyMzctft37Nu3e0pff6Yxmr24
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB8348
X-Spam-Status: No, score=-5.0 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 04 Dec 2023 15:06:24 -0800 (PST)


On 12/4/2023 12:53 PM, Tony Luck wrote:
> With Sub-NUMA Cluster mode enabled the scope of monitoring resources is
> per-NODE instead of per-L3 cache. Suffixes of directories with "L3" in
> their name refer to Sub-NUMA nodes instead of L3 cache ids.
>
> Users should be aware that SNC mode also affects the amount of L3 cache
> available for allocation within each SNC node.
>
> Signed-off-by: Tony Luck<tony.luck@intel.com>
> Tested-by: Shaopeng Tan<tan.shaopeng@jp.fujitsu.com>
> Reviewed-by: Peter Newman<peternewman@google.com>
> Reviewed-by: Reinette Chatre<reinette.chatre@intel.com>
> Reviewed-by: Shaopeng Tan<tan.shaopeng@jp.fujitsu.com>

Reviewed-by: Babu Moger <babu.moger@gmail.com>


> ---
>   Documentation/arch/x86/resctrl.rst | 25 +++++++++++++++++++++----
>   1 file changed, 21 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
> index a6279df64a9d..15f1cff6ee76 100644
> --- a/Documentation/arch/x86/resctrl.rst
> +++ b/Documentation/arch/x86/resctrl.rst
> @@ -366,10 +366,10 @@ When control is enabled all CTRL_MON groups will also contain:
>   When monitoring is enabled all MON groups will also contain:
>   
>   "mon_data":
> -	This contains a set of files organized by L3 domain and by
> -	RDT event. E.g. on a system with two L3 domains there will
> -	be subdirectories "mon_L3_00" and "mon_L3_01".	Each of these
> -	directories have one file per event (e.g. "llc_occupancy",
> +	This contains a set of files organized by L3 domain or by NUMA
> +	node (depending on whether Sub-NUMA Cluster (SNC) mode is disabled
> +	or enabled respectively) and by RDT event.  Each of these
> +	directories has one file per event (e.g. "llc_occupancy",
>   	"mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
>   	files provide a read out of the current value of the event for
>   	all tasks in the group. In CTRL_MON groups these files provide
> @@ -478,6 +478,23 @@ if non-contiguous 1s value is supported. On a system with a 20-bit mask
>   each bit represents 5% of the capacity of the cache. You could partition
>   the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000.
>   
> +Notes on Sub-NUMA Cluster mode
> +==============================
> +When SNC mode is enabled, Linux may load balance tasks between Sub-NUMA
> +nodes much more readily than between regular NUMA nodes since the CPUs
> +on Sub-NUMA nodes share the same L3 cache and the system may report
> +the NUMA distance between Sub-NUMA nodes with a lower value than used
> +for regular NUMA nodes.  Users who do not bind tasks to the CPUs of a
> +specific Sub-NUMA node must read the "llc_occupancy", "mbm_total_bytes",
> +and "mbm_local_bytes" for all Sub-NUMA nodes where the tasks may execute
> +to get the full view of traffic for which the tasks were the source.
> +
> +The cache allocation feature still provides the same number of
> +bits in a mask to control allocation into the L3 cache, but each
> +of those ways has its capacity reduced because the cache is divided
> +between the SNC nodes. The values reported in the resctrl
> +"size" files are adjusted accordingly.
> +
>   Memory bandwidth Allocation and monitoring
>   ==========================================
>