Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5;
Message-ID: <755ed028-f73a-47ed-a58a-65f4f48eaee3@intel.com>
Date:   Mon, 6 Nov 2023 09:03:24 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 23/24] selftests/resctrl: Add L2 CAT test
To:     =?UTF-8?Q?Ilpo_J=C3=A4rvinen?= <ilpo.jarvinen@linux.intel.com>
CC:     <linux-kselftest@vger.kernel.org>, Shuah Khan <shuah@kernel.org>,
        "Shaopeng Tan" <tan.shaopeng@jp.fujitsu.com>,
        =?UTF-8?Q?Maciej_Wiecz=C3=B3r-Retman?= 
        <maciej.wieczor-retman@intel.com>,
        Fenghua Yu <fenghua.yu@intel.com>,
        LKML <linux-kernel@vger.kernel.org>
References: <20231024092634.7122-1-ilpo.jarvinen@linux.intel.com>
 <20231024092634.7122-24-ilpo.jarvinen@linux.intel.com>
 <8051f3ef-1126-41fb-b6cc-f48441936dd7@intel.com>
 <2514e73e-2419-7c88-3f22-469db4b2fa25@linux.intel.com>
 <48c6795b-554a-4019-bb8d-a2ca0f6fbb2b@intel.com>
 <4008929-d12b-793e-dce8-eb5ba03b4ebb@linux.intel.com>
Content-Language: en-US
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <4008929-d12b-793e-dce8-eb5ba03b4ebb@linux.intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bDFWR1pmd1dzMUF3UHpYdlU2NnA0eXZXM0luZlpPZUpyRFFmbElub29xcEtr?=
 =?utf-8?B?dEdldXROSkZIc2pEZDFtY240RUVoVWZlNy9acUpXc3ZBTzlJN2puZEtNTVk4?=
 =?utf-8?B?VDJwNUpEelNraWtPbVJoMGVYOVlZL3dSdlREZ29BTWlmcTZjbzEya2hGN1Rt?=
 =?utf-8?B?VS8xeGFOYmhZSjBKZUkxTjhTa3VkOURYRitqdnhkd0xVa1N3cUxuSjRMN2dI?=
 =?utf-8?B?OE5SWW1WYlRDalE0RnJCNXF5Uk12M3FSVFhVSC9Td3drcTNycU4zSVcvNVV5?=
 =?utf-8?B?QlorMklJY0J6WndJVTlKK2hYa3M3WVdPUTRXL05nUUwzbStaVGJDekc4cldO?=
 =?utf-8?B?OEZqYjRkQktoMjA3MWhnRjd2U1JiMXpVTTVlMnBRdzdybzBHYnlTVUNGVzlv?=
 =?utf-8?B?SVJkVzRZWFJMQmVwK2dSczN0d2JKZzl1Q3RmVUhiWmlrMys0VmJwOVc3c2pV?=
 =?utf-8?B?SWZCNnpyMWJSZi9zMXh3RDV1ZDdLODFXNTR0bmVoM3BkNzJqUVhVZzIrOXUw?=
 =?utf-8?B?YVlhT3BPamtaTkw0UGFLdzUvSmhXSXZPK2I1NGhrUG05MUxjZEZuZTlFQWdp?=
 =?utf-8?B?UUNzNUFISFhrLy9kYkNwS1picS81RU5yWUV5aUgxTEpLakQwbkVRUklnSGFw?=
 =?utf-8?B?a28ySkhJMGhwajZZQllPNS9BMno5bTdvQjNZV3lTNTBKMGFuWm42NDJVUDls?=
 =?utf-8?B?aWtRYlR3dTJMclFkUlpiYWhNU3lhRWE3SkNXanFhdHJEd2xpYVZWTnQwYmMw?=
 =?utf-8?B?cUxNWnpTU29yUXFuNEFQd1V3RE1Qa2pKeFNYL1NrenpUUUdqREE3YTZpQnpN?=
 =?utf-8?B?N1ZNTkhnU2FZYmcrTVQxQnI1OXZzUzhFd3dBd29pVkdhWlZkb05Dc0pJRSsr?=
 =?utf-8?B?aUZhY1JUSlhLNVNoUUJ2cjZJenNrRE45NHEzZ1pqaEo4L1hjL0R3WGtDcXBQ?=
 =?utf-8?B?enh6VUYybjZWNFIwWmdLeUNjT1ZWbGNETjBja2NkbXA0a0N2bmsva2gzRWEv?=
 =?utf-8?B?NUVUcGpKbVlma2lsUTM4eUczVWR1cnBYRU5vMGwwbkxIRjU5eU1kdXE0TmNP?=
 =?utf-8?B?SjlLTXk5WDV1cC8zM2hQeU9yV1RWZkNuYU45clJFTW5YN3RkbkwrcUgzeXd2?=
 =?utf-8?B?SzhIVmV5WjdBTldncFFJN1NHbHhaME1MTTQvNXZtMWVOcFFzN1FhQmpaVmpF?=
 =?utf-8?B?VHNBYkRxc003cWhWNUQ3U05tME9ySEtqemJxcjk2d0RPc3RSOHU2UUk3MlJi?=
 =?utf-8?B?MFdyMmF2Mk0zWERYd3VLSVlYYVhxTUpoSGZUVzdSMjVYNytpR0ZoYnZIUFNO?=
 =?utf-8?B?eFJVR25UQ3l1Y0ZHVEFFdjFkTythT1hXc3NEVHBwWXVMQ0hXdmNpTkkrT0NB?=
 =?utf-8?B?akprOENZekZjdWRYZC9lWnpROTRBQ1JpRDRBYXFCdjJBYVNXb25QMGhJQU44?=
 =?utf-8?B?cGcrcTBXTjRTK3luSUxrUy9uWnNLN2tIQmNoUzJkTzJ0UUdwV2Zqb09MVGVM?=
 =?utf-8?B?aG9zM3oyNDcwWXBJUzRTUTFtS1pqWG1UeGNSdldvM1hweGxLMnRMU2NvSmVT?=
 =?utf-8?B?R3A2ODBsZ2w5VUJhV1VzYXJ4d0xaVzBpWWc0UjAvY0ZOUmdiSk9RNTNBVm1S?=
 =?utf-8?B?dTdYVmpPNUlkTEEyTnA0N01FVjNUV1hmZVI1MDNtUlQ2YkV1Z0ZkQXZYVDZN?=
 =?utf-8?B?Y0lwUTZpVm1JOXcyekptSlFUWEoxMHJmaEF4RlNoaEtxY0QrNmFSeXRNcGpw?=
 =?utf-8?B?dE9rOXBPaHNmMWxrc3ZMMHA0QjdBK1FheFkvRmNpTWh6TXFrMzJQREQwNUVY?=
 =?utf-8?B?K3VsZjhJZDF0dzd2QWJ2UnNMNlR1Zi8xREhLU3J0c1l4VFhZWG9Ma2huaElO?=
 =?utf-8?B?TEpOQkpOQ3ZEQVEzM1dESHRpdjg2YWgrekVSVy9RcXpKSUJweW11L1RPZkFs?=
 =?utf-8?B?ZkYxNjZIclMwWEhlNGFjbGx5RFg3SnJTUVAyT3R6UVA0Rnh1SFBOU2RVUlQ4?=
 =?utf-8?B?QmRuMm1TbVRldlBrR0d0YitOaGJFdnZYclpxKzZIT2RLbFdiSm5qNTZ5dTVB?=
 =?utf-8?B?Q3dYRzNCa0UzNGpDUnRGZHhMOFhLc2FmS3EyZjk5am9ua0JVM3oxendrOTZE?=
 =?utf-8?B?R1Z4VldtY3R4S21HMXdWQXdCVmhiTllVM3diTW9VMWp6SEN0cmdUV1RhbHE4?=
 =?utf-8?B?bHc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a8ecac54-f35a-4f18-e2e4-08dbdeea4ab5
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Nov 2023 17:03:26.2165
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 5d40XEmSDN04zS2CcGaTBaxOa7LMHhKXyXhZ1JycHyZqSLKUszowaidfXwSfKPHHGArITVZNyOXh8pCoPLunPcYqpKP/APQDmc6/qsm8bzM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW5PR11MB5788
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Mon, 06 Nov 2023 09:04:42 -0800 (PST)

Hi Ilpo,

On 11/6/2023 1:53 AM, Ilpo Järvinen wrote:
> On Fri, 3 Nov 2023, Reinette Chatre wrote:
>> On 11/3/2023 3:39 AM, Ilpo Järvinen wrote:
>>> On Thu, 2 Nov 2023, Reinette Chatre wrote:
>>>> On 10/24/2023 2:26 AM, Ilpo Järvinen wrote:
>>>
>>>>> Add L2 CAT selftest. As measuring L2 misses is not easily available
>>>>> with perf, use L3 accesses as a proxy for L2 CAT working or not.
>>>>
>>>> I understand the exact measurement is not available but I do notice some
>>>> L2 related symbolic counters when I run "perf list". l2_rqsts.all_demand_miss
>>>> looks promising.
>>>
>>> Okay, I was under impression that L2 misses are not available. Both based 
>>> on what you mentioned to me half an year ago and because of what flags I 
>>> found from the header. But I'll take another look into it.
>>
>> You are correct that when I did L2 testing a long time ago I used
>> the model specific L2 miss counts. I was hoping that things have improved
>> so that model specific counters are not needed, as you have tried here.
>> I found the l2_rqsts symbol while looking for alternatives but I am not
>> familiar enough with perf to know how these symbolic names are mapped.
>> I was hoping that they could be a simple drop-in replacement to
>> experiment with.
> 
> According to perf_event_open() manpage, mapping those symbolic names 
> requires libpfm so this would add a library dependency?

I do not see perf list using this library to determine the event and
umask but I am in unfamiliar territory. I'll have to spend some more
time here to determine options.

> 
>>>>> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
>>>>> ---
>>>>>  tools/testing/selftests/resctrl/cat_test.c    | 68 +++++++++++++++++--
>>>>>  tools/testing/selftests/resctrl/resctrl.h     |  1 +
>>>>>  .../testing/selftests/resctrl/resctrl_tests.c |  1 +
>>>>>  3 files changed, 63 insertions(+), 7 deletions(-)
>>>>>
>>>>> diff --git a/tools/testing/selftests/resctrl/cat_test.c b/tools/testing/selftests/resctrl/cat_test.c
>>>>> index 48a96acd9e31..a9c72022bb5a 100644
>>>>> --- a/tools/testing/selftests/resctrl/cat_test.c
>>>>> +++ b/tools/testing/selftests/resctrl/cat_test.c
>>>>> @@ -131,8 +131,47 @@ void cat_test_cleanup(void)
>>>>>  	remove(RESULT_FILE_NAME);
>>>>>  }
>>>>>  
>>>>> +/*
>>>>> + * L2 CAT test measures L2 misses indirectly using L3 accesses as a proxy
>>>>> + * because perf cannot directly provide the number of L2 misses (there are
>>>>> + * only platform specific ways to get the number of L2 misses).
>>>>> + *
>>>>> + * This function sets up L3 CAT to reduce noise from other processes during
>>>>> + * L2 CAT test.
>>>>
>>>> This motivation is not clear to me. Does the same isolation used during 
>>>> L3 CAT testing not work? I expected it to follow the same idea with the 
>>>> L2 cache split in two, the test using one part and the rest of the 
>>>> system using the other. Is that not enough isolation?
>>>
>>> Isolation for L2 is done very same way as with L3 and I think it itself 
>>> works just fine.
>>>
>>> However, because L2 CAT selftest as is measures L3 accesses that in ideal 
>>> world equals to L2 misses, isolating selftest related L3 accesses from the 
>>> rest of the system should reduce noise in the # of L3 accesses. It's not 
>>> mandatory though so if L3 CAT is not available the function just prints a 
>>> warning about the potential noise and does setup nothing for L3.
>>
>> This is not clear to me. If the read misses L2 and then accesses L3 then
>> it should not matter which part of L3 cache the work is isolated to. 
>> What noise do you have in mind?
> 
> The way it is currently done is to measure L3 accesses. If something else 
> runs at the same time as the CAT selftest, it can do mem accesses that 
> cause L3 accesses which is noise in the # of L3 accesses number since 
> those accesses were unrelated to the L2 CAT selftest.
> 

Creating a CAT allocation sets aside a portion of cache where a task/cpu
can allocation into cache, it does not prevent one task from accessing
the cache concurrently with another.

Reinette