Received-SPF: pass (google.com: domain of linux-kernel+bounces-107951-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <8ee6f553-16c0-4097-b5d8-af1598d1b85a@intel.com>
Date: Tue, 19 Mar 2024 10:51:14 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v16 0/9] Add support for Sub-NUMA cluster (SNC) systems
Content-Language: en-US
To: Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>, "Peter
 Newman" <peternewman@google.com>, Jonathan Corbet <corbet@lwn.net>, Shuah
 Khan <skhan@linuxfoundation.org>, <x86@kernel.org>
CC: Shaopeng Tan <tan.shaopeng@fujitsu.com>, James Morse
	<james.morse@arm.com>, Jamie Iles <quic_jiles@quicinc.com>, Babu Moger
	<babu.moger@amd.com>, Randy Dunlap <rdunlap@infradead.org>, Drew Fustini
	<dfustini@baylibre.com>, <linux-kernel@vger.kernel.org>,
	<linux-doc@vger.kernel.org>, <patches@lists.linux.dev>, "Wieczor-Retman,
 Maciej" <maciej.wieczor-retman@intel.com>
References: <20240312214247.91772-1-tony.luck@intel.com>
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20240312214247.91772-1-tony.luck@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Q0hKZU1tck9PMlltTzZ0ODVNQ3JIZ1dnSnpROEorYjdxalJuTHI2Z291L3g2?=
 =?utf-8?B?Y0ZhRDdQWEs3NUk0RE1FUFNTVHE3SGF2NkFUbHh2UE80K201bDMyYllYWEEw?=
 =?utf-8?B?bVBNR1A0QldOUkpET0xIUVBtcU9PS3N3QzdpbTdqL0lUTUNVSUlPa3ZDbzlX?=
 =?utf-8?B?cG5XZVdJclRRb2VYc1BPajdxZ1JJemUwWmxUdGhGYmE1dTc1aUtWVHl3U1Fv?=
 =?utf-8?B?Qkxhd2pld2MyNGxTN3dJeTlzRXB3WGlZV1k3THBiTkFzR1hiL1dBSHVBT0pZ?=
 =?utf-8?B?S24waG9kR0NkdGtpcFZJcEx1L09vbldEREo4dUU1K09WSFVsdU93c1dKUytr?=
 =?utf-8?B?WWpZcHJ2NUZrMk55S3E3L2U5TVUxd08zN3p5SGpJY3JQNjFxWTBzWUd5bWdG?=
 =?utf-8?B?b0RveXh3T3lpOE9RV0hVSldqNU95TDYxRmYxNFpoKzFHbXU4VW1qNGJxVHJP?=
 =?utf-8?B?Nk9BME1FdkwvLzVWSFJNR2RBaFFVNVNaQ1kwVUUzQ1k2aDZjdmR1T01kU2FC?=
 =?utf-8?B?a2NuOHQrd0NkNjczakgxQ0RjZlN2L2JveVFoajc4SEhNZEZOam5uVDFXdlVK?=
 =?utf-8?B?R2dSaEZYb3Z6eE9lQUo3bmZyL1drOXg2WHE3ZVZhTnpZWGY2YythS2hpSGhn?=
 =?utf-8?B?SWtNTE40RjlwV2N6UHkzYVhiend4aGpkK2t3aHo5VE9lRDVVVG1hNGlxZk9t?=
 =?utf-8?B?MEw2aThiMjRJcms3SHlRcXM4RDhnWUQwTUhMaDJ0MUtzWkhHNVBNYllwTU1F?=
 =?utf-8?B?M1RSdWhXVzluRENxYmZLYmFBOThEVC9Ec1hkU1FUWlhKcTJQZFYvNDdTcFNB?=
 =?utf-8?B?MWlDb1B3TENZOXRqY1kzMjl0ZG9vSTdNcWdUMU1NN2lmdUtWSmx1OWNqNXdT?=
 =?utf-8?B?d2ZqWjhUT3h4RWZqbXVaUmNpMldRbzVPMEZKYjFvK1dHUkFobTg5N1lNV0ZK?=
 =?utf-8?B?RG9uejRBRFBCZVJrVXJEc1ZaZkhtc2VoYjBpREY0a1Q1ZG5LQ2pqMTRXQkVz?=
 =?utf-8?B?cDhPRTVYaW4wZVJsU29rQjI2QUFBb0pIMVBPTEZsSXljNnRBOXlvRU82OXNN?=
 =?utf-8?B?eHdNVUR1RS9nd01YOGFvVW96dk1TdEN1cDRVbE5waTR3bnN6a3AyYm5iaXFq?=
 =?utf-8?B?Si8rYVA0cjhTTklFbWQ1TmhmbklkZ1pYbnFrVW5JbU5yRFVYbUdLRGltSU9t?=
 =?utf-8?B?cEppKzREQnRzWmhrc0hERldwOFZEcktVQUlTSitrMHFRVkhOM25PZjhDK1po?=
 =?utf-8?B?dnBFT09xd00zcExsUVAwVUdQaS9RWHlGNUplSkV3S0NyRlIrcVllWWRCWnc4?=
 =?utf-8?B?RFVYME1zM1hWTkU4Z3MrTEV4a3NFQlowK2ZsZk5hN250anJTR0N2YUViZ1V4?=
 =?utf-8?B?MjZvMTQrV25hRkptZWxWblJkSnJ3UGNCMTMyTVByL3hOZFFBcUIvSnM0d3ZI?=
 =?utf-8?B?UTNWTlpxd0V4VWlRMnBlS1FudDRBYm1YZHdEczdwaXI0YmQvUTVSbzdZMEZa?=
 =?utf-8?B?bStZZ1I3TW01MWN4RHczNE5EUjZpdWZLQ21ZTUp0UkhzZEkvSXE1ZEZhYUpR?=
 =?utf-8?B?LzRScEZoNzB0elNINlpIbTlKTzY4bTR2N2E4bU0wWlNBVDdFNE96T3BJQTN6?=
 =?utf-8?B?Z1pObGxBeUpteFdVUFYvSnI5U1JnYVVKVWI4ajdHRDlWWkRuVWN4TWRkVTdC?=
 =?utf-8?B?YzZMQnpOdXZDQjRVcXZxbW56QXR5dEM5UUJ1b1B1d2tEUTgwZFJpOTJnWE03?=
 =?utf-8?B?VmpZZk8zLzZ4Umh5UUdWZXgyZlMyRlFKZTlJcXFzY1l5SWU5S2c5TG4yYW5n?=
 =?utf-8?B?bVdSdHJteEliWUYyK1U2TUxMcXFaTlVOQXVNUGVtY2NxUlVHaWRyQmpXOTU4?=
 =?utf-8?B?ay9uRnEvY2J4cE0rYWxRdVl0TjNPdlA2dUQyeFczcGRYanRXYXgvakwvcXFN?=
 =?utf-8?B?M1JIWHN6K3hGY1Y1N1RrU0E2L0RKUDV1d0ZxU2Q5Z25qbndVNTk3TjlLRmE1?=
 =?utf-8?B?WW00eGVqSjRCeU1sbUlQeFczQkVFSGllMnNWdmpCV1R4ZmpFa2lHN2h1RnJL?=
 =?utf-8?B?ZmRmcitxd0l4QmxqTE9kNDJjQTY4NW5EeG5MQjhQd2lBcmhzOTMzY3RrTVVY?=
 =?utf-8?B?MTFkL3JSNnIwRmJRWEVqVk5kR1l6Q1JjL2dLenZTLzNxcUN6VzloUlNvSFJz?=
 =?utf-8?B?bWc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f1592324-0bcd-4799-71d6-08dc483d2c6f
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2024 17:51:15.6891
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: DURj8Fg09ec7pySKsy3TcYGSC5PkIbPeCE3W6tcRl6WjikInmB9eMtthiovDvl4LH+i8IHIbp2gsx4UBqx4qOdX8oKxTsVzleegIFUaHELQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR11MB7617
X-OriginatorOrg: intel.com

(+Maciej)

Hi Tony,

(Please add x86/resctrl to Subject prefix of cover letter)

On 3/12/2024 2:42 PM, Tony Luck wrote:
> The Sub-NUMA cluster feature on some Intel processors partitions the CPUs
> that share an L3 cache into two or more sets. This plays havoc with the
> Resource Director Technology (RDT) monitoring features.  Prior to this
> patch Intel has advised that SNC and RDT are incompatible.
> 
> Some of these CPU support an MSR that can partition the RMID counters in
> the same way. This allows monitoring features to be used. With the caveat
> that users must be aware that Linux may migrate tasks more frequently
> between SNC nodes than between "regular" NUMA nodes, so reading counters
> from all SNC nodes may be needed to get a complete picture of activity
> for tasks.
> 
> Cache and memory bandwidth allocation features continue to operate at
> the scope of the L3 cache.
> 
> Signed-off-by: Tony Luck <tony.luck@intel.com>
> 
> ---
> Changes since v15: Link: https://lore.kernel.org/all/20240228112935.8087-tony.luck@intel.com/
> 
> 0) Note that v14 Reviewed/Testing tags have been removed because of the
>    extent of refactoring to catch up with upstream. But nothing
>    fundamental changed, so everything should look familiar.
> 
> 1) Refactor to apply on top of Link: https://lore.kernel.org/all/20240308213846.77075-1-tony.luck@intel.com/
>    [So base commit is either tip x86/cache, or upstream current merge PLUS
>     the two patches in that series]
> 
> 2) Add patch 9 which adds files showing mappings from domains to CPUs
>    Reinette suggested this, James thinks it duplicates information
>    that can be gathered from /sys/devices/system/
>    Discussion here: Link: https://lore.kernel.org/all/ZetcM9GO2PH6SC0j@agluck-desk3/
>    This part is a nice-to-have. I'm fine if just the first eight patches
>    are applied without this while the discussion continues.

I agree to drop patch #9. 

The core support for SNC continue to look good to me (I just had a few nitpicks).

What remains is the user interface that continues to gather opinions [3]. These new
discussions were prompted by user space needing a way to determine if resctrl supports
SNC. This started by using the "size" file but thinking about it more user space could
also look at whether the number of L3 control domains are different from the number
of L3 monitoring domains? I am adding Maciej for his opinion (please also include him
in future versions of this series). 

Apart from the user space requirement to know if SNC is supported by resctrl there
is also the interface with which user space obtains the monitoring data.
James highlighted [1] that the interface used in this series uses existing files to
represent different content, and can thus be considered as "broken". It is not obvious
to me how to "fix" this. Should we continue to explore interfaces like [2] that
attempts to add SNC support into resctrl or should the message continue to be
that SNC "plays havoc with the RDT monitoring features" and users wanting to use
SNC and RDT at the same time are expected to adapt to the peculiar interface ...
or is the preference that after this series "SNC and RDT are compatible" and
thus presented with an intuitive interface?

Reinette

[1] https://lore.kernel.org/lkml/88430722-67b3-4f7d-8db2-95ee52b6f0b0@arm.com/
[2] https://lore.kernel.org/lkml/SJ1PR11MB608309F47C00F964E16205D6FC2D2@SJ1PR11MB6083.namprd11.prod.outlook.com/
[3] https://lore.kernel.org/lkml/SJ1PR11MB608310C72D7189C139EA6302FC212@SJ1PR11MB6083.namprd11.prod.outlook.com/