Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Message-ID: <3fb83b18-c9cc-42f6-813b-c5cfa526e91c@intel.com>
Date:   Mon, 20 Nov 2023 14:23:29 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v11 6/8] x86/resctrl: Introduce snc_nodes_per_l3_cache
Content-Language: en-US
To:     Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>,
        "Peter Newman" <peternewman@google.com>,
        Jonathan Corbet <corbet@lwn.net>,
        "Shuah Khan" <skhan@linuxfoundation.org>, <x86@kernel.org>
CC:     Shaopeng Tan <tan.shaopeng@fujitsu.com>,
        James Morse <james.morse@arm.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Babu Moger <babu.moger@amd.com>,
        Randy Dunlap <rdunlap@infradead.org>,
        <linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
        <patches@lists.linux.dev>
References: <20231031211708.37390-1-tony.luck@intel.com>
 <20231109230915.73600-1-tony.luck@intel.com>
 <20231109230915.73600-7-tony.luck@intel.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20231109230915.73600-7-tony.luck@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?em1ySC94UG5BZXFOVS9NeEJ1MlV0L3JFVzgrY0lXNXA5NitDWmNQR0FpNkpa?=
 =?utf-8?B?MDlDb290emZDMWhYcFQrYndwcGp3YVhTYkVCck1GZ1NRVFNOK0x2M0k3VlZE?=
 =?utf-8?B?SXdqaG9MR0MzRHdEbEIxb3hYSEN5VGpRMGFvZHpZT3RodkJNTmgwWm5mZ0hB?=
 =?utf-8?B?SWhPQjkzR096SnBQWTBqYW8xSHJjdG5xeFRSd0JEYXcvMnRBMVo4alZtVWMv?=
 =?utf-8?B?N0dpMklGc21aM3NIQWhlL3pSa1pmUWR4LzFPMEY5UEpBUURyOFlmTVhhVUtY?=
 =?utf-8?B?NGhkM1gvMkYybzB1M0dTc3YzdWJDT0dIYkpRS2xiYmtLc1N1MkdRaEVrQmdV?=
 =?utf-8?B?VGdEY0d6WkI1K0JMeU15eU5PWG9aQzVNaVZ5d2JGVlNXL0xwTlFTcDZtTmpi?=
 =?utf-8?B?MmpxTlVwdDg2U09LcHJmNkJWbm1EbDJyeW9zb1drNDZmSGRlLzYwM0svblI0?=
 =?utf-8?B?dnFwSUZ3RHV1YjByUFR2RXNJM1JpODJMNmRsN04waVpYUCtLK2ZIa2RpUFpR?=
 =?utf-8?B?SElSb3VBd2dHUloyZm9hQVJ2NDFrdEV2RU8zcnJ6Z1ZtK3I4TmJTTlpORjdI?=
 =?utf-8?B?L1RBSTZCK0NhSUdGS2pmb1hPOGIzcEs1S0h5ZnlCTHBwOWhUVkQyOFNZYUpB?=
 =?utf-8?B?ZUVDMlNMcWt6cDJCQlo1a29KdXB1ZlkyaEpucVRLNE12K2FPTUlYTnNpSmRC?=
 =?utf-8?B?TlpaNVUyTVVhcEU0WCtjMVVjVnZSNTBzVFM0RmthSUh1M0dXZ1BOd0VrcXEx?=
 =?utf-8?B?MlVTbDhnZXFhSWsrSTg4bEhHZkhYdXpyVnZza0V4ZktyQzk2NEdhQkloYm5w?=
 =?utf-8?B?WndpRnVNYzQ1bUJoNDZzN0FYOWFpTTBGdnB0ZDMvMVhuT0dhY0Vpa2o4SEhr?=
 =?utf-8?B?QkplNmUyZDRoVU4wVkhqUlNXOE4xQWtkS01VeWpwcWlxcHJQY3Q0T25kMWor?=
 =?utf-8?B?Y0Q1a2V6TXA4THhSTENJTHIyVThDMW50dkNSYzFLMURHdWM0VFY4eWRCak1S?=
 =?utf-8?B?SDVZZ2NreW9qWDEzdzlhUUcxeldqdmxmSU1MQmorWTU2UEJQR0tTZ1h6dUxr?=
 =?utf-8?B?SThLUklWWGlhWVRSZ0JLZHJjM3ZmQzhQdUhrSjVhZk5MRTNKdHludUtiQTd4?=
 =?utf-8?B?YnJCRDZpUXBQYkh5MWhKTUYrQmQ1ZU1XWGNDUm9pK1JyMFJmMjZjM1B6dWFl?=
 =?utf-8?B?OTVWT21SYnpFNEhja3NBOWJXSHhpVWtBcitIRkVRZ2ZOMXh3WW9QZnZ6dUVm?=
 =?utf-8?B?QmRxcXFzOFJ2Zk8zZGNNQ1g0NU9xSWE4SHNCR2oxaTdwN05rWFNKbmFhSDZV?=
 =?utf-8?B?MHZBNno1YTgzQWxyYVp2anpwNDE1TjRjNE5MUjVLV3plbDdaS2FXMS9sM0Ur?=
 =?utf-8?B?NXBacHJvRS91M3pYQUtTK2tmcURaNHZxT3NEUUVyQ2V5R3dnczZBYTlvbnFQ?=
 =?utf-8?B?M1ZsalJTMWpBY2txemZXTUJpM05zZjRaWUR2NjZDVUNCOFlSZ0RqSzRTa2oy?=
 =?utf-8?B?VmtxbEJVYzdZNFF2RUV1QlEyVVJzaW9yMGozTnducTZETkdvYXFjeE1MWFFr?=
 =?utf-8?B?ZVpabGpmU0c3cTlWU2F1d1JWbElrMTEyQkZwSFByckQ4VmVWSzJ5dUplT0JK?=
 =?utf-8?B?ZGVKOHVKajhnNVp0bVJ0Y1FtdFI1TVVGVVRZa0FNeTVwbXNYbVM2UmJadVRq?=
 =?utf-8?B?MDNVakNXM0N1ZEJDcjJOWVZEajZOWEtQY2UxeDJQTWtrSUJLU2M3SzRyL1Ev?=
 =?utf-8?B?b1VsMjZnTCtzZTNZbzk4MGhOdkM4RUVMR1BhUFNLN1dPQ0JtYXZzaFRTUG4y?=
 =?utf-8?B?bHNQY1oyN0Y3Ymw3TVh1NUJGRTlDMlFyRWNsVWZRN0VGOU4xVjhUMGpjV3hw?=
 =?utf-8?B?WTVvTzI1YW9Ndnc4c3Y1dzh5REJVLzR0ektPS2VER1Z1TWxRNUJPL3lOUGdD?=
 =?utf-8?B?SU5QdiswclNVcHFJbk9hRWUvZSszVGo4ZWw4K2U3N09ZcWovck5LLzJqbHor?=
 =?utf-8?B?dU9KMWQ4c2FYM2MvWEh3Skl5SUNpNVloaEo1eWQ2dmRBZTQ4bTNZbzFVQzZW?=
 =?utf-8?B?c0FFYkpOV0dDdjBZT1BaWTl5OGVjV0dJeUozazVzUEViSDNMZnVzMDJwR2ky?=
 =?utf-8?B?dytGVHZVa2FhdEltaklUOHJWQkp6M1VjTURrV3hJeVZKUXF0RGNUMEJpSElO?=
 =?utf-8?B?Z0E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 4379f6b0-48e6-447c-ea12-08dbea17532b
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Nov 2023 22:23:30.4988
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: sjY7Koo9XqL7mo79IzhG2Sv+PBdgXOQ7UO88yLi1Qi/WI51wb4MEQtn4JVHEPA0owhpRJAg5mJ4tckRvpRpPPHEcg+hRvDNgf755lM1IdXo=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR11MB7797
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 20 Nov 2023 14:23:56 -0800 (PST)

Hi Tony,

On 11/9/2023 3:09 PM, Tony Luck wrote:
> Intel Sub-NUMA Cluster (SNC) is a feature that subdivides the CPU cores
> and memory controllers on a socket into two or more groups. These are
> presented to the operating system as NUMA nodes.
> 
> This may enable some workloads to have slightly lower latency to memory
> as the memory controller(s) in an SNC node are electrically closer to the
> CPU cores on that SNC node. This cost may be offset by lower bandwidth
> since the memory accesses for each core can only be interleaved between
> the memory controllers on the same SNC node.
> 
> Resctrl monitoring on an Intel system depends upon attaching RMIDs to tasks
> to track L3 cache occupancy and memory bandwidth. There is an MSR that
> controls how the RMIDs are shared between SNC nodes.
> 
> The default mode divides them numerically. E.g. when there are two SNC
> nodes on a socket the lower number half of the RMIDs are given to the
> first node, the remainder to the second node. This would be difficult
> to use with the Linux resctrl interface as specific RMID values assigned
> to resctrl groups are not visible to users.
> 
> The other mode divides the RMIDs and renumbers the ones on the second
> SNC node to start from zero.
> 
> Even with this renumbering SNC mode requires several changes in resctrl
> behavior for correct operation.
> 
> Add a global integer "snc_nodes_per_l3_cache" that will show how many
> SNC nodes share each L3 cache. When this is "1", SNC mode is either
> not implemented, or not enabled, but all places that need to check
> it are updated to take appropriate action when SNC mode is enabled.
> 
> Code that needs to take action when SNC is enabled is:
> 1) The number of logical RMIDs per L3 cache available for use is the
>    number of physical RMIDs divided by the number of SNC nodes.
> 2) Likewise the "mon_scale" value must be divided by the number of SNC
>    nodes.
> 3) The RMID renumbering operates when using the value from the
>    IA32_PQR_ASSOC MSR to count accesses by a task. When reading an RMID
>    counter, code must adjust from the logical RMID used to the physical
>    RMID value for the SNC node that it wishes to read and load the
>    adjusted value into the IA32_QM_EVTSEL MSR.
> 4) The L3 cache is divided between the SNC nodes. So the value
>    reported in the resctrl "size" file is divided by the number of SNC
>    nodes because the effective amount of cache that can be allocated
>    is reduced by that factor.
> 5) The "-o mba_MBps" mount option must be disabled in SNC mode
>    because the monitoring is being done per SNC node, while the
>    bandwidth allocation is still done at the L3 cache scope.
>    Trying to use this feedback loop might result in contradictory
>    changes to the throttling level coming from each of the SNC
>    node bandwidth measurements.
> 

The latter part of this changelog stopped being in imperative mood.
To reduce confusion I slightly reworked it below to address the
parts I noticed. I often get it wrong myself so please check again.

	Add a global integer "snc_nodes_per_l3_cache" that shows how many
	SNC nodes share each L3 cache. When "snc_nodes_per_l3_cache" is "1",
	SNC mode is either not implemented, or not enabled.

	Update all places to take appropriate action when SNC mode is enabled:
	1) The number of logical RMIDs per L3 cache available for use is the
	   number of physical RMIDs divided by the number of SNC nodes.
	2) Likewise the "mon_scale" value must be divided by the number of SNC
	   nodes.
	3) The RMID renumbering operates when using the value from the
	   IA32_PQR_ASSOC MSR to count accesses by a task. When reading an RMID
	   counter, adjust from the logical RMID to the physical
	   RMID value for the SNC node that it wishes to read and load the
	   adjusted value into the IA32_QM_EVTSEL MSR.
	4) Divide the L3 cache between the SNC nodes. Divide the value
	   reported in the resctrl "size" file by the number of SNC
	   nodes because the effective amount of cache that can be allocated
	   is reduced by that factor.
	5) Disable the "-o mba_MBps" mount option in SNC mode
	   because the monitoring is being done per SNC node, while the
	   bandwidth allocation is still done at the L3 cache scope.
	   Trying to use this feedback loop might result in contradictory
	   changes to the throttling level coming from each of the SNC
	   node bandwidth measurements.

> Reviewed-by: Peter Newman <peternewman@google.com>
> Signed-off-by: Tony Luck <tony.luck@intel.com>
> ---

(same comment as previous patch about commit tag ordering)

Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>

Reinette