Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35;
Date:   Tue, 31 Oct 2023 08:24:04 +0100
From:   Maciej =?utf-8?Q?Wiecz=C3=B3r-Retman?= 
        <maciej.wieczor-retman@intel.com>
To:     Ilpo =?utf-8?B?SsOkcnZpbmVu?= <ilpo.jarvinen@linux.intel.com>
CC:     <linux-kselftest@vger.kernel.org>,
        Reinette Chatre <reinette.chatre@intel.com>,
        Shuah Khan <shuah@kernel.org>,
        Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>,
        Fenghua Yu <fenghua.yu@intel.com>,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 16/24] selftests/resctrl: Rewrite Cache Allocation
 Technology (CAT) test
Message-ID: <o4tsktjvcey7wjb6pkrp7fp7v3m2sgqqhsuzgsdyigypr2tlxa@me4fl5mjrhig>
References: <20231024092634.7122-1-ilpo.jarvinen@linux.intel.com>
 <20231024092634.7122-17-ilpo.jarvinen@linux.intel.com>
 <kq6qds2hgcg3vlgokgyr4lukm7weuj3thvl7p2panfmk72ovpy@nshm6iva5wfr>
 <fb5e1a50-ba7-1ee8-8bf2-bb8b64b27b1@linux.intel.com>
Content-Type: text/plain; charset="iso-8859-1"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <fb5e1a50-ba7-1ee8-8bf2-bb8b64b27b1@linux.intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?W7ZYVhy7EV67P9EplVjtauzm+jWR5KDNIHVqV1q/wWwAa90QjjkfpTlnMF?=
 =?iso-8859-1?Q?VQSqfAD6CFs5e3Ki9mUsaQyJHyB5M7v4pSh4IwTADSV+fjFqx/yRuK2ZU6?=
 =?iso-8859-1?Q?NjAn3f5V/fnIslcK+6YinAdjh0tlYmORmt25GXX7R3OuZIeLvv39NduE6P?=
 =?iso-8859-1?Q?fuvkeynDsoSNrZyyBELMt0KDblUOJdcWR2+ahfbkxtlkqNxIONnb95hIWb?=
 =?iso-8859-1?Q?Mei9o5KXXR1Y8yRnzbSzYzWsIh9CQLO0k526fqQw/8TsVvFf4lVbxVU2Sg?=
 =?iso-8859-1?Q?lVVnvAkR0whTrmr7ApKohfHnmIR7BfNrgntJDORDihuhDQDKG/iblCWmZm?=
 =?iso-8859-1?Q?IAOiBmqqZmCSHmTjVz30KHEx+8IDG0tDR7wmhFcphHA10h5X1ZBiigN8p9?=
 =?iso-8859-1?Q?5WGdR+kfQKFzemgJU0Cent32189Gcb+J3Jdod/HuyUnPJeEKqD5IdP+jXT?=
 =?iso-8859-1?Q?rvXcFcNuW8v2wdB3CwNh/EFhFi3MmmVXVjPQd35XPQDMwcAN+nfzIZ1oYu?=
 =?iso-8859-1?Q?RS9Aed5C1SCdAOOCAXL0kskJ9SwFRoNLQkxPZNDt9l4mKYZwH9P3ECNWQ9?=
 =?iso-8859-1?Q?/KQyXZJkUGxuzChCmO2EWc6lAvpa+XYkT37SsR53CbY0TCJiodau3jnIKx?=
 =?iso-8859-1?Q?/TdvKtTTydBkH0tR7HdKwGMIade5vEj9bKXqCKNOAn8fnm3iBjTdzAXWpb?=
 =?iso-8859-1?Q?3JYz3gKR/oq+ClI6Q8RbberRJL0HpYp07EhnEVqdmeFvbxzNUseXodAGUS?=
 =?iso-8859-1?Q?Jaf0UQ/uBRhi//p7AuSTpjac8OaZ4k/PeMzhFNdxN1T7vx6H25DJcGmaDr?=
 =?iso-8859-1?Q?P8+UStCJBWTdOA25uAry5ILO59au2PCOhs+D7zj1Y3jsmjRwAwvqfTn3p8?=
 =?iso-8859-1?Q?ewPSMlNxNkRU95XszGbfZ7LROyXltM0j09gVL01Ik7c8pnVDqn61C6h6gi?=
 =?iso-8859-1?Q?hMax7Us8pwohTYQRmccxXAxBjWPfXsiqq1/Mm+yBG0pgkz45NBS+HcfN9e?=
 =?iso-8859-1?Q?xY90FMVlnx1p0cGWxGikMVvuy1eB0RkYGOu4YIg/NxAoMVSQi0ten0hbMj?=
 =?iso-8859-1?Q?GSt86GNZkXpfb2JDoGcb2oqEWdxzF4y01OhM+xC6wWMy4FOObRjO1TIOlF?=
 =?iso-8859-1?Q?W5IGC3kzM7/le1u4ZkJUPLaQAjFQrZ5SU9Nk3NPRKy0ZrnsgGAYt8o0//1?=
 =?iso-8859-1?Q?mHVkoYAfXa7A1RjezVIo8llg+SwrjS8wAcg+bxU1dT1JbxM+T1TGXTG0zH?=
 =?iso-8859-1?Q?G0T7G5zgyEuQmbgNohTXVPnTCBiqIaeHqQnH0d2hwRe2uT8Fkavc1um9QX?=
 =?iso-8859-1?Q?m3tDHO5Ql62nq7uoKOvvQEVhQ01PhE73z3BF/sU3v27+I4rVYc79aVyTy8?=
 =?iso-8859-1?Q?N8YwVYCkfXode+AGJF1M8/S3eEdtgn4epovVZ+NOjKiPeEoqPWkYORp42Y?=
 =?iso-8859-1?Q?xiwQ+g4Zn5UfUs65OUhAgHEvMLwnrZGhFPX7qflr8aAHKp3dh8McpJBEfj?=
 =?iso-8859-1?Q?IdoxfVM6enCtnmTFWYfWZJXf24spkhpsWFu84Y4O1ZmyfINviuG3jQCOQa?=
 =?iso-8859-1?Q?3UjtzBH4OB7xxOMP/6tlZMjLF4kYfQNaPSxOH71Tpg4Vxr7uq4dyq/asYp?=
 =?iso-8859-1?Q?kYnhG15w3Nw89t+cFQH4BOkY2fW1zknfqLSEuIq12wvyans4C+rujalTKm?=
 =?iso-8859-1?Q?Q/QMiTpJaAn+BYVDpvY=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 9f0c2cc8-2fe5-46b0-fe18-08dbd9e25f05
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6231.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Oct 2023 07:24:08.4932
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: SMCGmFVriiMaTzYC3rZPUBNTaZ/YWVh1KQAB187YhmlSKiJT0JH/flYz8p/ahN/zsEmr8lsvF2ad12FQCyIhP1nQy/d3s3O6lqpDsROrYDs=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR11MB8343
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-1.3 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Tue, 31 Oct 2023 00:24:29 -0700 (PDT)

On 2023-10-27 at 15:32:58 +0300, Ilpo J?rvinen wrote:
>On Fri, 27 Oct 2023, Maciej Wiecz?r-Retman wrote:
>> On 2023-10-24 at 12:26:26 +0300, Ilpo J?rvinen wrote:
>> >-	ksft_print_msg("%s Check cache miss rate within %lu%%\n",
>> >-		       ret ? "Fail:" : "Pass:", max_diff_percent);
>> >+		ksft_print_msg("%s Check cache miss rate changed more than %.1f%%\n",
>> >+			       ret ? "Fail:" : "Pass:", (float)min_diff_percent);
>> 
>> Shouldn't "Fail" and "Pass" be flipped in the ternary operator? Or the condition
>> sign above "<" should be ">"?
>
>I must not touch ret ? "Fail:" : "Pass:" logic, it's the correct way 
>around. If I'd touch it, it'd break what the calling code assumes about 
>the return value.
>
>(More explanation below).
>
>> Now it looks like if (avg_diff * 100) is smaller than the min_diff_percent the
>> test is supposed to fail but the text suggests it's the other way around.
>>
>> I also ran this selftest and that's the output:
>> 
>> # Pass: Check cache miss rate changed more than 3.0%
>> # Percent diff=45.8
>> # Number of bits: 4
>> # Average LLC val: 322489
>> # Cache span (lines): 294912
>> # Pass: Check cache miss rate changed more than 2.0%
>> # Percent diff=38.0
>> # Number of bits: 3
>> # Average LLC val: 445005
>> # Cache span (lines): 221184
>> # Pass: Check cache miss rate changed more than 1.0%
>> # Percent diff=27.2
>> # Number of bits: 2
>> # Average LLC val: 566145
>> # Cache span (lines): 147456
>> # Pass: Check cache miss rate changed more than 0.0%
>> # Percent diff=18.3
>> # Number of bits: 1
>> # Average LLC val: 669657
>> # Cache span (lines): 73728
>> ok 1 CAT: test
>> 
>> The diff percentages are much larger than the thresholds they're supposed to
>> be within and the test is passed.
>
>No, the whole test logic is changed dramatically by this patch and 
>failure logic is reverse now because of it. Note how I also altered these 
>things:
>
>- MAX_DIFF_PERCENT -> MIN_DIFF_PERCENT_PER_BIT
>- max_diff_percent -> min_diff_percent
>- "cache miss rate within" -> "cache miss rate changed more than"
>
>The new CAT test measures the # of cache misses (or in case of L2 CAT 
>test, LLC accesses which is used as a proxy for L2 misses). Then it takes 
>one bit away from the allocation mask and repeats the measurement.
>
>If the # of LLC misses changes more than min_diff_precent when the 
>number of bits in the allocation was changed, it is a strong indicator CAT 
>is working like it should. Based on your numbers above, I'm extremely 
>confident CAT works as expected!
>
>I know for a fact that when the selftest is bound to a wrong resource id 
>(which actually occurs on broadwell's with CoD enabled without one of the 
>later patches in this series), this test is guaranteed to fail 100%, 
>there's no noticeable difference measured in LLC misses in that case.

Thanks for explaining. Looking at it again the patch makes sense and seems very
coherent.

-- 
Kind regards
Maciej Wiecz?r-Retman