Received-SPF: pass (google.com: domain of linux-kernel+bounces-197133-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <07325558-c816-42ec-9f74-464df2f98e9f@intel.com>
Date: Fri, 31 May 2024 09:17:46 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v2 1/2] selftests/resctrl: Adjust effective L3 cache size
 with SNC enabled
To: "Luck, Tony" <tony.luck@intel.com>, "Wieczor-Retman, Maciej"
	<maciej.wieczor-retman@intel.com>, "Yu, Fenghua" <fenghua.yu@intel.com>,
	"shuah@kernel.org" <shuah@kernel.org>
CC: "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"ilpo.jarvinen@linux.intel.com" <ilpo.jarvinen@linux.intel.com>
References: <cover.1715769576.git.maciej.wieczor-retman@intel.com>
 <fe9295c6be677d187b1607185e23993dbfe74761.1715769576.git.maciej.wieczor-retman@intel.com>
 <9fa47acf-86b1-4602-8790-39ed80fd775a@intel.com>
 <SJ1PR11MB60833CF8BD209555A5370299FCF32@SJ1PR11MB6083.namprd11.prod.outlook.com>
 <992cd5db-1cc8-4238-879b-8ef0406edc33@intel.com>
 <SJ1PR11MB60835941880FC77E1B1E9FEFFCFC2@SJ1PR11MB6083.namprd11.prod.outlook.com>
Content-Language: en-US
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <SJ1PR11MB60835941880FC77E1B1E9FEFFCFC2@SJ1PR11MB6083.namprd11.prod.outlook.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QmJVOWhMNy9NS1E0Tm5RdDhvRk1oRlF0dUNxNUt4MHpIQ1R6KzE1Nm5RRmZK?=
 =?utf-8?B?R0lXVjFjRGRFSFdXZzIxblZ5cHlYd3FUL2Vnb2pUcm9FYnhoV3VrZm5BeHpt?=
 =?utf-8?B?aHpSSHUwc25tbW51MThZWnhGQ3F2VHFlbUV5Ukx2M0tIWDVvT1Y4VnBoTzho?=
 =?utf-8?B?ODhuRXB3UlF1MWRjRzlvam56SzZRVG14OVFzR3FhMzlJSUJmcVpnNnJOZ3Na?=
 =?utf-8?B?QnBvd01KcDFpWk84ZDdKcW91NnNNazMxYXBsbFM3VGc5Y3FqUzI1S0xJR1pv?=
 =?utf-8?B?T1JLcTcrMzBmaHZMTXZJVkdGMG40WnMyZEU4WGxYZThTZDFLeVlpUTFtVnNr?=
 =?utf-8?B?SE1CYkoyTVVWUkVSZkxjc1hmb0tnZmd3V2JtNDdkbmh0YUcrTllVUmtPSVk5?=
 =?utf-8?B?NU1rVVdKdDlFN3lXNCtQZEUyMUg2U2ZtRmRPbDZhQkd2ZlhIY1dHcTltRUxS?=
 =?utf-8?B?Z1lBYVFIQmJHTFRWdlhVQTJPRXkxczJlVmpGLzdYakQ5bVRLdGlMK3RDNDEv?=
 =?utf-8?B?U2wxenIrVG5BUVFPTFpabmpkOFQ3SVZuaXZRZUhXNTJ3dTRRZWZvQVF4Lzcr?=
 =?utf-8?B?Vm84ditsWS9IYk5uazFEUS9vZzh1bnRZMnZwN2UvdWMwbzFRdnhsalgvQkpP?=
 =?utf-8?B?NmNPTVlRaWN3S1M2MXlPS29hYThsTUpLOFVLbE9QTkx1dE9IaWg3MEN2ZmFu?=
 =?utf-8?B?OFVyMnJ4amtLQVJ3c2VCakYwQjFlS0VmYzVxRjlZVDFGK2hKZCtzVzNsN2t4?=
 =?utf-8?B?MHdLTU85QmJWQml6cXlqNHNwT3ZOTnlNK3J1aEc4L2R5eTFLbjRKeEFwd0J2?=
 =?utf-8?B?aVgxY2dwNFhqb2xRa2tvbGZTR2ViWldWaXhmRXVaM0c2bGo4KzR6bGV0RlIy?=
 =?utf-8?B?OWJwbFhoY05vWGt3Y3BLNnFkNHZLbjhLT1VEbU9Nd3ZlMjZJNVlIM01KVFhY?=
 =?utf-8?B?ZGdBWWFXRFhjOG5mVmFrbkQvMUkrK1RhakZpRHh0dUdpSVFaNGk1Y3l4bllK?=
 =?utf-8?B?ZXV3V3p5aWxvcVd4QThwdEJDSFc4Zkd1VlRBSGd1V0ZtL2poWkpQUVFKRHRL?=
 =?utf-8?B?R00vZXpLa3JyZzZpandXcmFMbHJoSTVyblV4d0dHVU5OY1ZVM2ppYlJyNGxY?=
 =?utf-8?B?eXVGMVVLZmEvYlNNNFVhMkhTTzg5N0g5Nmo2Ukg4cWE4UXk2bFJWQVR2dkVs?=
 =?utf-8?B?L2xneGVLRmg5dlYxcUNkRW9HWDlOc2ZYV0ZWL2lKV3pXd0ZVTDd4ZFpDTmNh?=
 =?utf-8?B?T0FqT1lFODVhNG1oekNIajNscndDMXVvbXFJazhtNVhkM2wwcmZZMVo0d3U3?=
 =?utf-8?B?QldVU0xIY0QrMVhFaDNWUG1uUnFmUk5HQnBudnA5N1lSc1dEaFZWcmNmM3RS?=
 =?utf-8?B?aW51NytOWDBXREd6VmFTRCt2M0QxeWFxK3BUSDg1dkJjZFpGZmU3TFM1V1l3?=
 =?utf-8?B?U2hoV3VpdmN5bzdtMTQySHphOHZTSDhNRHl2M3pMVWdJcEtvVmhzMThkQk4y?=
 =?utf-8?B?SFFxanoxejBRVlp3dmJ2UHc5NTJjZXJXK2ZQQTJIYWN2L2VsMFBQV3kxd2kz?=
 =?utf-8?B?bC9odFY1QlBXRVJXcEU5NGdDa2h6NVpvYmR5dmErckpnSDZBZmE2VDZlck02?=
 =?utf-8?B?b0tNSDFySzhGbHJHNFJXdzZTSkJWQnBrQlZFZ3NtS3RlbHFicWFGaDVMVjRS?=
 =?utf-8?B?ZENzTkh3RWNnaWdLVTEvQUtEcFJyL0ZqYkowRlZBMXVMMVExd0RtZ2lGYWto?=
 =?utf-8?B?NTh5Y1ppbXNtUHYzWG5RTVNkelhBMllkZ3h4WExoUXhlRkR2cUU3SzlCTS96?=
 =?utf-8?B?clJyeFFGTWJsTVlTWXNNN2swSHZBNDVOWjM5b2MvNnF0MDNQMnVMSTBoSzBy?=
 =?utf-8?B?OXVjNjV6QlJqQWdtaDVHcmozUk4rcFRPbTMvWHc4NU42cG1xOW0xSlBHOGJk?=
 =?utf-8?B?alhtWEZZWDVWRjYvam9xOFJYMDAxQU5EN0dRSlduMWVMMTN5bm9uUUo0dVhR?=
 =?utf-8?B?dVpvWmJyQU0rNmVzdFlTL1cvQ05lQXQ4UTU0ZFF2bjE5K3lXK3VkV1RGTDRC?=
 =?utf-8?B?dDhDT3F1d0Q5NFVBVkZDSnhBNENFZDVrd1ZNaTdzOG84OE1VVEVtdG9nMTY0?=
 =?utf-8?B?UHpvV2QvS1RMaXljU0FubEl1Y1NaN20zM1FRcmYvSGRNWlZRdFZZT29MelZq?=
 =?utf-8?B?Z0E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: efb21aef-c31d-469a-1e08-08dc818d368e
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2024 16:17:48.6491
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /86e0fJobXSa5E4cnQbBd4B52a21EdKwO3hWhAVeE5RJCQUCYbilOopLMS9Ob3HsvBFVaNuKrqwlxt2luwTYPFvxJ3+qVThmQxaS7A/fhWg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB6845
X-OriginatorOrg: intel.com


Hi Tony and Maciej,

On 5/30/24 5:34 PM, Luck, Tony wrote:
>>>>> When SNC mode is enabled the effective amount of L3 cache available
>>>>> for allocation is divided by the number of nodes per L3.
>>>>
>>>> This was a mistake in original implementation and no longer done.
>>>
>>> My original kernel code adjusted value reported in the "size" file in resctrl.
>>> That's no longer done because the effective size depends on how applications
>>> are allocating and using memory. Since the kernel can't know that, it
>>> seemed best to just report the total size of the cache.
>>>
>>> But I think the resctrl tests still need to take this into account when running
>>> llc_occupancy tests.
>>>
>>> E.g. on a 2-way SNC system with a 100MB L3 cache a test that allocates
>>> memory from its local SNC node (default behavior without using libnuma)
>>> will only see 50 MB llc_occupancy with a fully populated L3 mask in the
>>> schemata file.
>>
>> This seems to contradict the "Cache and memory bandwidth allocation features
>> continue to operate at the scope of the L3 cache." statement from [1]?
> 
> I'll clean that up. MBA isn't affected. But cache allocation is affected in that
> the amount of cache represented by each bit in the masks in the schemata
> file is reduced by a factor equal to SNC nodes per L3 cache.

Thanks Tony. I trust that this is what Maciej intended since the change
is specifically named "Adjuct _effective_ L3 cache size". I'd like to
recommend that your comments be added before the change to
get_cache_size() ...

	/*
	 * The amount of cache represented by each bit in the masks
	 * in the schemata file is reduced by a factor equal to SNC
	 * nodes per L3 cache.
	 * E.g. on a SNC-2 system with a 100MB L3 cache a test that
	 * allocates memory from its local SNC node (default behavior
	 * without using libnuma) will only see 50 MB llc_occupancy
	 * with a fully populated L3 mask in the schemata file.
	 */

Reinette