Received-SPF: pass (google.com: domain of linux-kernel+bounces-109165-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Wed, 20 Mar 2024 17:27:24 +0100
From: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
To: Reinette Chatre <reinette.chatre@intel.com>
CC: Tony Luck <tony.luck@intel.com>, Fenghua Yu <fenghua.yu@intel.com>, "Peter
 Newman" <peternewman@google.com>, Jonathan Corbet <corbet@lwn.net>, Shuah
 Khan <skhan@linuxfoundation.org>, <x86@kernel.org>, Shaopeng Tan
	<tan.shaopeng@fujitsu.com>, James Morse <james.morse@arm.com>, Jamie Iles
	<quic_jiles@quicinc.com>, Babu Moger <babu.moger@amd.com>, Randy Dunlap
	<rdunlap@infradead.org>, Drew Fustini <dfustini@baylibre.com>,
	<linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
	<patches@lists.linux.dev>
Subject: Re: [PATCH v16 0/9] Add support for Sub-NUMA cluster (SNC) systems
Message-ID: <uk4k6ayim6yfcquwl7wwc6mrfta23suefid2t7pxldlsm7zs2f@4yw6hm2m4z6p>
References: <20240312214247.91772-1-tony.luck@intel.com>
 <8ee6f553-16c0-4097-b5d8-af1598d1b85a@intel.com>
 <5vnrboxpkrriiqcbwrxtbextu6pduqjfchlvsdn5nwe7lr7vou@dngffvu7a5fh>
 <7656a3fe-56d2-4a9f-9dfd-6f2666f38110@intel.com>
Content-Type: text/plain; charset="iso-8859-1"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <7656a3fe-56d2-4a9f-9dfd-6f2666f38110@intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?MHfLslfJGds7LeCQtBpEhv/XUNKDjUwd0Nqinph0/5PoxyrJmDrPVtQhkt?=
 =?iso-8859-1?Q?ix56hRxIVs2R7h0etSJ39MqnjY+a4PUxWsDXHwM4T2Gge9GzNZ4Fm9rz63?=
 =?iso-8859-1?Q?UEqlzeLeS9S94gldi0gINC9WhP8b3GE2Ood1N2Z4QUMXHYPhWdh0pDu9yS?=
 =?iso-8859-1?Q?QL03mpDkfajyDES8xG2IC5cfCFxkwUPBV38cHQV2sxl55tKixu3GbNl5zf?=
 =?iso-8859-1?Q?ImfDHhqetXNqHJC7ICkT71bBuX6woWwnfU8JlZ0jyDRmGCKkynYuyPF3OC?=
 =?iso-8859-1?Q?79TyOSleyoirzenI5EBlTwyNHrC3xHB5mD8+Jb/N6AgC7Le7TGM/u/58lZ?=
 =?iso-8859-1?Q?3Z4Xa6X0C4Zk8Gqi5M9GM65IY9jMY4139poGr7KX1CLdI4Zshn10LLOakj?=
 =?iso-8859-1?Q?cX7VauWTgX4hQ2UZ+pRF34vvpKTXiUu23MZ8oQKpr1ep/NYIBwFV2TzSWV?=
 =?iso-8859-1?Q?WtmJBardWcKRUhwyT3maWG/sm/cNRUPFUTjyUfWek0+md4zaeKxfVkOTQ8?=
 =?iso-8859-1?Q?2D/YXc9dLs9N7S25B+RZ8Wq/7xtiZ4nHAoFq8oTLe9rvKcB7laxZTvvJcq?=
 =?iso-8859-1?Q?+1Isl6hG1Gpjw1wkMnu4iPD4tFkFsB/hc8YFOC0XFpeNPo+bVhGSjn+QEt?=
 =?iso-8859-1?Q?k/hW1ySbnX7ALOaDb4LsL9UERy3VypNkcwrrajF9nAdISkPG6KDVerLSzy?=
 =?iso-8859-1?Q?XNDwIBD2YJDIuOh6oWLVA4Iad0zqs37Kyzk6EEouP3RYBTaceJK9RPG9Qf?=
 =?iso-8859-1?Q?xdXPm5asakccuWhsapwD0p71BRlrf7j4NF14c+OKz4AGFjNvDwK5YIBd/I?=
 =?iso-8859-1?Q?HWfe6FKXM4RruUmC331i0ZFzNUZQ6bApqY42A0rRt/g4iWtIJkExq88VpK?=
 =?iso-8859-1?Q?SHWNfvIoLiTWujXuwWAiuC67SWDjhcySlpzZUYBwAcqXRRj7sUfetiGy5y?=
 =?iso-8859-1?Q?4z+Iy6yR1MDy7k5uMR7rVcxnafrUnoVxT/QomSvMQ4EWiO/qTruLXhr2cE?=
 =?iso-8859-1?Q?wYu2Ac53WKN6W8BXsuFcKFlj7N1SAmB7lTVtN4Pq/EtpxJaDvh+6zAZuKK?=
 =?iso-8859-1?Q?R/6rYZZaXCCm9/+bLPIpt1SYbcDUSIIRq93y16kb0ABwXc52t3Vd1bSY15?=
 =?iso-8859-1?Q?Vyr11IDjDhNvGc7WO8ylvlAbQmTTFRklEBgrrev2GG7zHt8jBdv51hpP3+?=
 =?iso-8859-1?Q?TO4rEo9WWix5zDDd/xKgHyBly/sS+XqYTUdjucBYdNdqhmZL2K43ZJtByt?=
 =?iso-8859-1?Q?FwxqX8FrW/GSYPAKgbUllOk6K9mg0vDIl9mPO0IS4oeyY19hTHo8qWUnkt?=
 =?iso-8859-1?Q?yfeaP++Noz4XrRSv4d6i05gEkjVKDHvriLCN6rPQnYSz7dMquGntwdzN2j?=
 =?iso-8859-1?Q?XxCUuhiqtdGIGOfGaSEkAHB1oGefqRy43v10/tPQKhIKOq2MZtHBhrqEsg?=
 =?iso-8859-1?Q?v2bgRIln67/up73Suq8d4+bq8vne4NsBnXMi7nVFUGkDrwe98wCRG+n0VA?=
 =?iso-8859-1?Q?VTwQuawExR3DCbnBbokDiintk03sQ+V3O5SMYqetHB1T22MqiP4oCMw0Cm?=
 =?iso-8859-1?Q?mgcOOYt/0IemRiwtgbvBPeEuM1Cs0ph3/z1PYz9mJ39niuOceUVzL7Yv2o?=
 =?iso-8859-1?Q?shNO0WG94Mv9w7HMNgv5PijoCs8/wB5eWDSOERTUODb6ga3bIuus66HQnt?=
 =?iso-8859-1?Q?dY0qrSJsaOFIyBaTBiU=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: be0b40f4-692d-4c54-4c23-08dc48faa901
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6231.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Mar 2024 16:27:39.6518
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: b+drX+V9rhDM5B//jjAHlhJB18Ob11xV3vEl/2aPfyTVYGRkwKeffWBs4c+djaT7gA/JpxYB+x76Pu7dPIqxbtibS3tKh6kS/iUQlU9xtEM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB7570
X-OriginatorOrg: intel.com

On 2024-03-20 at 08:50:51 -0700, Reinette Chatre wrote:
>Hi Maciej,
>
>On 3/20/2024 8:21 AM, Maciej Wieczor-Retman wrote:
>> Hi Reinette,
>> 
>> On 2024-03-19 at 10:51:14 -0700, Reinette Chatre wrote:
>>> What remains is the user interface that continues to gather opinions [3]. These new
>>> discussions were prompted by user space needing a way to determine if resctrl supports
>>> SNC. This started by using the "size" file but thinking about it more user space could
>>> also look at whether the number of L3 control domains are different from the number
>>> of L3 monitoring domains? I am adding Maciej for his opinion (please also include him
>>> in future versions of this series). 
>> 
>> By this do you mean comparing the contents of main "schemata" file with the
>> number of mon_L3_* files?
>
>(assuming you mean mon_L3_* directories)
>
>Yes, the "schemata" file can be used. There is also the "bit_usage" file in
>the info directory that indicates how many control domains there are.

I just did a test on my IceLake server that has SNC enabled and you're right.
This can indicate kernel support for SNC. It also seems more reliable in
determining the ratio of nodes per socket (since the ratio of cpus per
cache/node can potentially fail with offline cpus).

>Do you think doing so also falls into the "not obvious text parsing and size
>comparison" category?

Ideally [1] seems just the most user friendly in my opinion. Comparing schemata
with mon_L3_* directories feels like a good next solution though.

>>> Apart from the user space requirement to know if SNC is supported by resctrl there
>>> is also the interface with which user space obtains the monitoring data.
>>> James highlighted [1] that the interface used in this series uses existing files to
>>> represent different content, and can thus be considered as "broken". It is not obvious
>>> to me how to "fix" this. Should we continue to explore interfaces like [2] that
>>> attempts to add SNC support into resctrl or should the message continue to be
>>> that SNC "plays havoc with the RDT monitoring features" and users wanting to use
>>> SNC and RDT at the same time are expected to adapt to the peculiar interface ...
>>> or is the preference that after this series "SNC and RDT are compatible" and
>>> thus presented with an intuitive interface?
>> 
>> I kind of liked this idea [1]. Hiding SNC related information behind some not
>> obvious text parsing and size comparisons might eliminate any ease of use for
>> userspace applications. But I agree with you [2] that it's hard to predict the
>> future for this interface and any potential problems with setting up this
>> file structure.
>
>Thanks to you for trying this out from user space side and highlighting the
>difficulty trying to do so.

I suppose it wasn't that difficult in execution but it required a lot of
thinking about what is a reliable way for checking kernel support, what can fail
in checking the ratio of nodes to sockets etc. Maybe once we have this figured
out all it will take is a good documentation and userspace applications won't
have a hard time with SNC.

>
>Reinette
>

[1] https://lore.kernel.org/all/SJ1PR11MB608309F47C00F964E16205D6FC2D2@SJ1PR11MB6083.namprd11.prod.outlook.com/
[2] https://lore.kernel.org/all/7f15a700-f23a-48f9-b335-13ea1735ad84@intel.com/

-- 
Kind regards
Maciej Wiecz?r-Retman