Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <eed9454b-a276-e545-64a7-5d236dd38d82@intel.com>
Date:   Thu, 27 Apr 2023 08:15:29 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.10.0
Subject: Re: [PATCH v2 24/24] selftests/resctrl: Rewrite Cache Allocation
 Technology (CAT) test
Content-Language: en-US
To:     =?UTF-8?Q?Ilpo_J=c3=a4rvinen?= <ilpo.jarvinen@linux.intel.com>
CC:     <linux-kselftest@vger.kernel.org>,
        Fenghua Yu <fenghua.yu@intel.com>,
        Shuah Khan <shuah@kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        "Shaopeng Tan" <tan.shaopeng@jp.fujitsu.com>
References: <20230418114506.46788-1-ilpo.jarvinen@linux.intel.com>
 <20230418114506.46788-25-ilpo.jarvinen@linux.intel.com>
 <eac72c7b-bd01-084e-cf23-ec5e8e0e3cf3@intel.com>
 <b2f72637-5c16-d25b-8a0-3fee4ca9ec7e@linux.intel.com>
 <ff2b3b61-57d2-04da-a719-c72d8bfd49eb@intel.com>
 <dfe24719-cd5e-9ff-9278-a92356dc1490@linux.intel.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <dfe24719-cd5e-9ff-9278-a92356dc1490@linux.intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cmJ4cy9xM25SYVNSRUNrYTFJd0wvSU1uS05jazVPbTk5ZnJ3Yk1sTlZQdnFQ?=
 =?utf-8?B?SGkzRmhIdzZuMWNMK2tGcTFobmJPdlJtUDNIRFBXTUs2MFlsTGtHUWllcUNM?=
 =?utf-8?B?N2t2b0ZBL0k3N1R6cm1scWI5dEowUjFwZGxPSzFEQTBHM3J4dzhxS2VzWHdO?=
 =?utf-8?B?bkYyNitFVngwZURHelNPcWE3anhZK2tHcVN6V0hhYnZMeG1xRkR2aURJdVpL?=
 =?utf-8?B?SHZuUkYvSHRUeEdGeUhCY04zY3lUUERYb21DYlkrR0xxVWI5Wm5Lb1dLVmxy?=
 =?utf-8?B?R3hMclJ1TGZhU2FPL3J1dmRhbDdabW1VZFVTUmYwSWpGUWFTQU9XTFF1MjJZ?=
 =?utf-8?B?djE2dWpoN0QwYS9YQmFwL015bTNVVVhBZUdRdlFCVjZURG9lVHdQeFJ4dmp6?=
 =?utf-8?B?OTFoVUFGUFV4TVZlRzFCaUwremlhRHdIdmd3SWtNWEVaOEsxUTB6a3BiVmdC?=
 =?utf-8?B?L3FWQlAvZk55d3kwMHVwZ1lwRUpYWm1NVlN1Q2pnODZrMitmL2pNbi8vVktD?=
 =?utf-8?B?Zks4V2xPenRDbUVxRjYxbllVTURBYmp3bUJ1L3pKQWhVSjdlZ2dQRXNrU1Nx?=
 =?utf-8?B?RVVUSG0wbGZCQU1KZ24rVG5YOXJnZWovWHphU1M1Rnh3bFlLdXg4ejBCZWNN?=
 =?utf-8?B?dGxnV0hNNXJucVcvUUtEdFlVYnNnbjZKMEMxYzM0R1BlNmo2Q0k4KzhjaG5F?=
 =?utf-8?B?R0tEdXgvekdERHNRb2I2Y21iZzdRMHJia0N0T3MvUXNydGw2L3BXQStyTjV6?=
 =?utf-8?B?MkZra0JSMVQvUElXWnJQWVBZQ1NpSkxVTHl4WWhEYmNzS21xc2xSZm9lQmh3?=
 =?utf-8?B?WVBNSlVvSWRsRVRJd3hYVFJNRk9mNXdnTUV6VndzVTNscVVjUXZYNEVGaTUw?=
 =?utf-8?B?MnVBOSt1ZzBCN2ZQbmMrbllRNEVSMVJaZW93eHVLdDFPL1lLSDBuVUdxZm1m?=
 =?utf-8?B?WVRFVTFZTmhlRDFYcTFYdTA1L2Fsci93R0tlMExkQnhKY1prdzNtOWdielB2?=
 =?utf-8?B?NVBudC9BaFNkY20wRjFxV1h3Z2xhdUQvN2l4d2NjVFNQRGJXVFcyOHRDZXA0?=
 =?utf-8?B?WGs1QUFSSDZSMWU1NEY2UUZHSFpoRW9QRmZCK3hhZXkvdHRKRGozcWVjT2Zn?=
 =?utf-8?B?d2w5QVZoVnRzMmRqdy9xVlQxbXhoUXcvRlpLZVJRWkxwZkJETUhXK2ZneFZp?=
 =?utf-8?B?UXdOdWJ4TDdiYk12SjN5bjJ4Nit4YnlBWVlZcHpOcE9kY1JZaTY2R2pwclJB?=
 =?utf-8?B?Vk8zeHFYMVpWM1VRbndMZWhLL2diUXd3WTRlRzMvT3dRWFdmK3ZTd3gxSUcz?=
 =?utf-8?B?b2h1M1NieXpoWTJCcFhFb1h2RVRJRmRUOXZ0czhzRG5PQUNDRG83cFNnNU9G?=
 =?utf-8?B?UnVISnYzU2hjV0Q3VysrYUhpSHpmTWZYWSt5eDBOMy8wbkxKOW9YQWxoL21J?=
 =?utf-8?B?a1ZhRVlVcjN6VEdBeE9YUERJOXFRRjJJT3RqQnNBbjNGRkFIamsrdFY2MC9s?=
 =?utf-8?B?djkyd0tSQitHMWR4SmMwWkEwekVzOVFVMFJKKzVTWXVFaUd2aWZ3Z2dMU0R4?=
 =?utf-8?B?UlQ4aGdrak55STEvck1rTFB6c3ROS1hKbHJJcU9xTzltMWFJL284ZlNGa3pp?=
 =?utf-8?B?dzVRVWY5endseEYrWVhEK1BFbW1IZWtZSjc5M2ZHL2l0MDBJMTBzTGRiNk5W?=
 =?utf-8?B?TXpObG14WHlYZnZqc2ZaMEhsMXRFTlQ0cHRjNjFtay9iT3ZMMjNOU00zM1VF?=
 =?utf-8?B?eXpBY3kwZm9pVEN2ZE0wVU4zK3BLT3NsalppOUFGS3JCdXVTOTdlb253SjVG?=
 =?utf-8?B?QW8rUjNXUjZqU2ZIVzBxWmkweTFmNHpJUVhIQnlTMjVMK0w4MTFqaEh0V1dZ?=
 =?utf-8?B?TWxVRWRjUHRSVHRETkRNZEV0L1ROcUpxOWtCd1lBU25oN2VmSmxQWXZoUVBy?=
 =?utf-8?B?aW5wcTlJaXNxU0w3ZlprZkFZZmV0RkpkaVJNdjlzNUtBNXlBT3FIVW9CeDRN?=
 =?utf-8?B?YWN6dUhsTTJIV0I5ajI4bW1QcFBPTWptMzhMbURTQzAxN0k3RHE0WWV4c2xO?=
 =?utf-8?B?UDQrSWFodGZxc0Yzc3hDTU1YaW0xdHN2NThFeDE1cDZrd3I1ditkQm4reU53?=
 =?utf-8?B?bFRET21HNmQxNlJuK2pBTThJN3kzckVXa0lhdlZzZGE2OCtHRGhhYzhBTDdk?=
 =?utf-8?B?N2c9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 74860142-1a5f-4978-1ef4-08db47323dfb
X-MS-Exchange-CrossTenant-AuthSource: SN7PR11MB7566.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Apr 2023 15:15:31.8825
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: clibbTWkL0UTmGejYF9p6Jp1C61kVF2NMuo8ZCLf2d+X1j6LbJeTHBcW1w2SYhE8tEt4PoIDx9Xytt5UqVAagKiXJx2YZ3HmIge8oJ6mTlA=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR11MB7967
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Ilpo,

On 4/27/2023 1:04 AM, Ilpo Järvinen wrote:
> On Wed, 26 Apr 2023, Reinette Chatre wrote:
>> On 4/26/2023 6:58 AM, Ilpo Järvinen wrote:
>>> On Fri, 21 Apr 2023, Reinette Chatre wrote:
>>>> On 4/18/2023 4:45 AM, Ilpo Järvinen wrote:
>>
>> ...
>>
>>>>> diff --git a/tools/testing/selftests/resctrl/cat_test.c b/tools/testing/selftests/resctrl/cat_test.c
>>>>> index 4b505fdb35d7..85053829b9c5 100644
>>>>> --- a/tools/testing/selftests/resctrl/cat_test.c
>>>>> +++ b/tools/testing/selftests/resctrl/cat_test.c
>>>>> @@ -11,11 +11,12 @@
>>>>>  #include "resctrl.h"
>>>>>  #include <unistd.h>
>>>>>  
>>>>> -#define RESULT_FILE_NAME1	"result_cat1"
>>>>> -#define RESULT_FILE_NAME2	"result_cat2"
>>>>> -#define NUM_OF_RUNS		5
>>>>> -#define MAX_DIFF_PERCENT	4
>>>>> -#define MAX_DIFF		1000000
>>>>> +#define RESULT_FILE_NAME		"result_cat"
>>>>> +#define NUM_OF_RUNS			5
>>>>> +#define MIN_DIFF_PERCENT_PER_BIT	2
>>>>
>>>> Could you please start a new trend that adds documentation
>>>> that explains what this constant means and how it was chosen?
>>>
>>> I can try although that particular 2 was a bit handwavy that just seems to 
>>> work with the tests I performed.
>>
>> The changelog claims that the existing CAT test does not work with
>> this new test offered as replacement. Considering that I do think it
>> is important to have confidence that this test is able to test CAT.
>> The words "handwave" and "seems to work" are red flags to me.
>> When merged, these tests will be run on a variety of platforms with
>> various configurations. Using test criteria based on measurements
>> from one particular system may work but there needs to be confidence
>> that the criteria maps to all systems these tests will be run on.
> 
> My "tests" (in plural) were not limited to one particular system but 
> included systems from different generations.
> 

Thank you very much for your thorough testing. Having this information
accompany this change will surely help to increase confidence in the
value chosen. 

Thank you very much

Reinette