Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <f6ffd978-7e3b-2dfc-1860-80046f9a7a1e@intel.com>
Date:   Fri, 11 Aug 2023 10:33:18 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.14.0
Subject: Re: [PATCH v4 6/7] x86/resctrl: Update documentation with Sub-NUMA
 cluster changes
Content-Language: en-US
To:     Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>,
        "Peter Newman" <peternewman@google.com>,
        Jonathan Corbet <corbet@lwn.net>,
        Shuah Khan <skhan@linuxfoundation.org>, <x86@kernel.org>
CC:     Shaopeng Tan <tan.shaopeng@fujitsu.com>,
        James Morse <james.morse@arm.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Babu Moger <babu.moger@amd.com>,
        Randy Dunlap <rdunlap@infradead.org>,
        <linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
        <patches@lists.linux.dev>
References: <20230713163207.219710-1-tony.luck@intel.com>
 <20230722190740.326190-1-tony.luck@intel.com>
 <20230722190740.326190-7-tony.luck@intel.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20230722190740.326190-7-tony.luck@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bE13eW5ER3NqTm1tVldmdE1MaTVqQ0NxYVpzY21Ca3RrVUFrRWM5NW4xMVN6?=
 =?utf-8?B?NHlFZ2x1YXY3ejdhYjFwdjBUWitVQzFhbmNpeWxkVThYd1RvelVaUEtxdUtT?=
 =?utf-8?B?K2NSU2J2ZVd6RnpLZkQvL3NGeDFJd2djT1l5TWJlQUNXd0Y0UTJCbk1ONUd3?=
 =?utf-8?B?WDREcmtPakxCU3VLWUZhK1c0cE5haWwwSEVlS25iTVJTQ0pvcmw0VkRoWk55?=
 =?utf-8?B?UEN0UFA0MFdWZW53THRWdE51MGJ1eVk2Zm9jREtUVll0Q05UK0JYOU8vZTRR?=
 =?utf-8?B?NEJlYWpOTW12R204bjdTQ3FHNEdMSmhlbERiNHNpYTBRUXNNaUJpR1laaWRi?=
 =?utf-8?B?R2tCWnBOOTJqLzQwRXk2RTh4V2t1YmY5bkdEN2oyZlRvN0xuN3hQQkErSGI4?=
 =?utf-8?B?S1B5SHAzRHJWa3kvWVN6N2V5cXN4Y1d2VW4yVTUyR0NnQjd5M2JoQjlVU0Z3?=
 =?utf-8?B?TzNuSktsQ1pBZGpzV3owdDl1ZTFpQlNyVXFlNDBIMlphMHR1RHE3YTl5M3RM?=
 =?utf-8?B?UE5XNEVaSTFHV21tOXdmd3lmeC9mdnhRemRybDJ4bWJSSy84cFdMWk1yWXln?=
 =?utf-8?B?a3l3QW8yblJ1Mld5b0hsdGs4U3VYSFdWV29Wa3BiS0IwOVJCUk02b0t5djAz?=
 =?utf-8?B?SjA1TGVVSXQwVWFJMFdQUjR4a1l3UmJnUTFkQ08wVFJoNisxZEtkNE40dTRD?=
 =?utf-8?B?YlhqbWxGSWY3d3Jtb0tmOEN4dkQ1VnQ4V3hiemgyTDVCQStRZkpvMk5BZFkv?=
 =?utf-8?B?dUZZS3hoS0hhY2hXbk52eStUamE3bVNNQlozYm02dk1xU2RvT0MxbmQyMmNy?=
 =?utf-8?B?WmRwRzFCNlFyOTB2RGs3M2VBSTdvREFLQnp3NXorcUR2V2xNVWtlQzBzM2k5?=
 =?utf-8?B?eDZSdVZBR25Ed2FzdjlPUE1OTGdpK2FyMXljeCt0aWlLaEVzY0pDTGdwMko3?=
 =?utf-8?B?VlpoRVdmeTJwSU9JQXp2MmZIeGt5YitEeEJlbFVNYzhmNUdrQkdiVWg5NFFj?=
 =?utf-8?B?MVFZSkhobDBxNmY5K3VaTFpIK082UTBvdTlVOGd3YmtISSsvLzRxai9iN3U0?=
 =?utf-8?B?blpLbnF6aVFOVzFkSHpia0xzc0UyUk1JYlJWZ1dBSlE5c0lUUWttVXRPSFlN?=
 =?utf-8?B?UThLblp1emtrbFFLeFd3SkwxOFU2K3RZakZqbWZuQ0h0ZnlXN09DTklhaWdS?=
 =?utf-8?B?YytzMzNGTkFZODlNc0JpbTZnZGsvME5wUWpJUThkSTQ4WUNnVVdCcGIrbktE?=
 =?utf-8?B?MUUycEVZWHg0a2RoL3JDTDBMVkd3UEJSelpwbDFSK0UzWjBucjB3QmViWEs0?=
 =?utf-8?B?OHFaajdWNWJScEhDdHNueHcyRmQ5KzVHVEtEeDNUNnVLZ0hoODhuTnFJY1oy?=
 =?utf-8?B?SVZqOWh4RE9HWFZVNGEyQ21lUHFyKzR6M09IY1lOZEgvaldPMlZWRk5JaVkz?=
 =?utf-8?B?bk5VVWNkdncySVlDc0N0NnJrcEczYmdvWUg2RzQwV2k3UTl0TGowQUdWbzN4?=
 =?utf-8?B?VHR4TTdoczJFVGJRQU9zTHM5T3VMNE9VUzkzaitnK0o1L2ord0Nwdy9JcEVM?=
 =?utf-8?B?VjdZdExBSnZBNk5FYkpwM1djcXNENUtHcy80OVgyUzVqSHBYZ0JIZ1NWZ2Vk?=
 =?utf-8?B?RzA2akxCVFdhcEpTZ0Nob0dTTlhWYk1kWm9nb3RBS3NaTkkxT0VlekNOcVJH?=
 =?utf-8?B?dEdqQ2NLS0gwZE4yWWlSdDM4ZGxNQTZVZHpKZk5pc2ZjUi91bFAybXVBZC9a?=
 =?utf-8?B?M3dwMk5sN2N0M1dkaUFtcWVwc2ZHWFl2L3gxTVVHR2VqSmh1YjJFelViY3Iy?=
 =?utf-8?B?cGhWM2VFS1BlT2wra250SCtadVBCVVByaVhJQWtTSjltcjBGcHNiT1IydUh2?=
 =?utf-8?B?bEd6T2ZxVVcvZm04ZFY5NWtjUVRJYmdzb01IMmF4REkyREY3UFdxWExYcU5z?=
 =?utf-8?B?ZGZLbW1TQkJGUUVCRTBSOW50RFc4Q2NSNjU0eTF6TEd2Z3lQYjdPZENPU2dv?=
 =?utf-8?B?ZDRadUVVV3d5NHZEV2dVTmxuQ3BSNWV4QVhDZzQxdVo4WHpScDBXaldlU3Jv?=
 =?utf-8?B?VjRaRXlhQW9CRUd3UFprQ2V4eWxGdS85VTRkRC9NV3hhNnNSTTlrTW5jcVNK?=
 =?utf-8?B?OVpKTkdkbU1DNGYwa0VVYzZGRmE3ejcvMmtoRkRhcDNZSDIvQWNMVXAyYnNP?=
 =?utf-8?B?eGc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 659cc0b0-1339-40fb-877f-08db9a910e7d
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Aug 2023 17:33:20.8813
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ZHNH4EZhLOTGTyT92leDb27lCDkBe7YfwwnThPFYMRfU/e4rPZtsiS6MmHrtS13p6W2N0peZVi2cyYlHw2KjfAP9iojhZSm0ZPe81EgP/2A=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5609
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,
        SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Tony,

On 7/22/2023 12:07 PM, Tony Luck wrote:
> With Sub-NUMA Cluster mode enabled the scope of monitoring resources is
> per-NODE instead of per-L3 cache. Suffixes of directories with "L3" in
> their name refer to Sub-NUMA nodes instead of L3 cache ids.
> 
> Signed-off-by: Tony Luck <tony.luck@intel.com>
> Reviewed-by: Peter Newman <peternewman@google.com>
> ---
>  Documentation/arch/x86/resctrl.rst | 10 +++++++---
>  1 file changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
> index cb05d90111b4..4d9ddb91751d 100644
> --- a/Documentation/arch/x86/resctrl.rst
> +++ b/Documentation/arch/x86/resctrl.rst
> @@ -345,9 +345,13 @@ When control is enabled all CTRL_MON groups will also contain:
>  When monitoring is enabled all MON groups will also contain:
>  
>  "mon_data":
> -	This contains a set of files organized by L3 domain and by
> -	RDT event. E.g. on a system with two L3 domains there will
> -	be subdirectories "mon_L3_00" and "mon_L3_01".	Each of these
> +	This contains a set of files organized by L3 domain or by NUMA
> +	node (depending on whether Sub-NUMA Cluster (SNC) mode is disabled
> +	or enabled respectively) and by RDT event. E.g. on a system with
> +	SNC mode disabled with two L3 domains there will be subdirectories
> +	"mon_L3_00" and "mon_L3_01". The numerical suffix refers to the
> +	L3 cache id.  With SNC enabled the directory names are the same,
> +	but the numerical suffix refers to the node id.  Each of these
>  	directories have one file per event (e.g. "llc_occupancy",
>  	"mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
>  	files provide a read out of the current value of the event for

I think it would be helpful to add a modified version of the snippet
(from previous patch changelog) regarding well-behaved NUMA apps.
With the above it may be confusing that a single cache allocation has
multiple cache occupancy counters. 

This also changes the meaning of the numbers in the directory names.
The documentation already provides guidance on how to find the cache
ID of a logical CPU (see section "Cache IDs"). I think it will be 
helpful to add a snippet that makes it clear to users how to map
a CPU to its node ID.

Reinette