Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33;
Message-ID: <721c6735-dab9-49c7-bdb2-b34388144e21@intel.com>
Date:   Fri, 3 Nov 2023 15:53:41 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 24/24] selftests/resctrl: Ignore failures from L2 CAT test
 with <= 2 bits
Content-Language: en-US
To:     =?UTF-8?Q?Ilpo_J=C3=A4rvinen?= <ilpo.jarvinen@linux.intel.com>
CC:     <linux-kselftest@vger.kernel.org>, Shuah Khan <shuah@kernel.org>,
        Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>,
        =?UTF-8?Q?Maciej_Wiecz=C3=B3r-Retman?= 
        <maciej.wieczor-retman@intel.com>,
        Fenghua Yu <fenghua.yu@intel.com>,
        LKML <linux-kernel@vger.kernel.org>
References: <20231024092634.7122-1-ilpo.jarvinen@linux.intel.com>
 <20231024092634.7122-25-ilpo.jarvinen@linux.intel.com>
 <a5160a37-7778-49ff-8c25-92846d42e87e@intel.com>
 <6426a6be-5e92-6c5d-7025-6308c1f3c24@linux.intel.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <6426a6be-5e92-6c5d-7025-6308c1f3c24@linux.intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UWlWWFdicGJZSWc3bThwUzIraDlXQ1JhMXJ3VGFJbXhzTlhKQWxMVVpEamVK?=
 =?utf-8?B?TTRSd3d3aTVnV08xQ2cycmVxR2FlUVZGSFhGUWU3STduVTl3RVlMQmYwRUFT?=
 =?utf-8?B?T2k1VTUvRlcrU2ppTks3eGI0U0E0cFJFSGx0RWhhTkRVSnVtdTBHcUJHZjJT?=
 =?utf-8?B?c05ZNXFCOXB2dUtaOWxIb1hFNnE2RmhzamRDTzQxbEFsY1E2bXFocWNZaEJW?=
 =?utf-8?B?bi9BRFBTYno2QnRrZS9MNnlnc3BDZUQrZVpHMkxYbjhpcXZUbGYzRGRhUkE0?=
 =?utf-8?B?RE9oVm0zaGk2dnY1OG5CSElKYjV3MVRNWE9yUEFnVmlzTjhrQ3JBU2NTQUNB?=
 =?utf-8?B?YUp3R2hZZ28ySUFldjZOTElaWEg1RmxFV3cwZnBJVDdBVDU5dnpjU1hKbERi?=
 =?utf-8?B?Z01tVTc2U1UzZFU2VGRoWFBabTc1ZjZhTGRFeFpobmVrQldwOE13eDBFcFZG?=
 =?utf-8?B?VFV3amwrdzM0WE5vbW5JcTcyQ1dtOE92V044SzFSVG5oc0Fsa2d3cVkxRENN?=
 =?utf-8?B?czhBWUJQYWZIUFRWWTU5VDM2RHZkV0F3MGY2VDBUR2dvZmFDS3BHNGRSakpL?=
 =?utf-8?B?TEduNGpGdGZ3ZG5kWDV5UjhEdEFBbHljaE1GWkZISXpLUmh2V0d3bHBRY2VG?=
 =?utf-8?B?QThaYkEzRVo4bk9TNGFhcmxhOUF6cjd4MXlBVytxVlAySUZzS3Rvb3BJaVRu?=
 =?utf-8?B?cUdvRUYwdGJUVGMrWUJHWHdFZFhmZzlMVUViUHNNOFpHYndqVE1XbHlSeTV4?=
 =?utf-8?B?MGVBOFlrYWIwd2E5M2VudGJoY1c3T0hwamhReWp6TGV4RUpJY1Qrc0QzL0JL?=
 =?utf-8?B?bXlPQk92VHQ1WnMwTGRFN3F5a0NhRVFoQTNRdlJFcTE5Q2IvdE5Fa0hRVDJs?=
 =?utf-8?B?TnZJdnR1QlZncjRTcFlKcmoyK082anR0UkhPa3NhdENwTFFnVzE2U2pTRUR6?=
 =?utf-8?B?VFlQakRrUGw0STNYVlJmVGkxQzJqUXl6dThUWTdWdnI3QjZ4MzBBS1I2dFU0?=
 =?utf-8?B?c2lRUWhWRHdtd2lHL1NtdE80NWd6dk5ZSE5RRkJhRkhycCt3T2h1Y1NITkF0?=
 =?utf-8?B?b25rQ2xFNkZPcWlsT3NMOXY1dDBwcnljaGhtNXY0OEI4cWxHRVNRWmRsa3pM?=
 =?utf-8?B?bGUxc3VXWG1sRzMwUExWTzRXdVZwTEFZaDdlVWZiZHRuRjZoV3JkaE1lNUhU?=
 =?utf-8?B?SE1qN20vL3hSK2ZrbHUydmtNOW5rRlAvZlJ0K2Z5OFJtZURDRTIzOEFlb0h3?=
 =?utf-8?B?dUE1WCt4STdOblAzODlSZ2d2d0ptalRRdlR3UldSZVRWaWQ2VER4SzNtbHps?=
 =?utf-8?B?Rm14cU83bENnYmdreEZiWVFDMFU3UkFkdnVGSWFxNko3MDhNUHBadmlNeE05?=
 =?utf-8?B?THhMLzhBK0pyWHF4eTBMMVNLRk1HNjJ4Y3NxaW4vZFZ6Q2RlOVV3NHEzVDJT?=
 =?utf-8?B?V05wK3V0NTdpeUxjV1Q3QVcyRlBZWE9HSGtWWG5kdEhQQjZ2QmxDTDhvb21p?=
 =?utf-8?B?RUdWM0hBcDB3ZGE3SjJhTDV1MGpGZmpXUWNRTnUxWjd2WmhsQkdkQTljcXhs?=
 =?utf-8?B?RnVHVFlGN3Ava09KY1hKQ3daRXFzOWJ4L2tsZVdFZGhoZUsyNnFIcFYydG9E?=
 =?utf-8?B?dnVxMURDd0dwQXd0Z0cyYzNKanFSUTZ1OElMWVVzUC9nK2I0SGgwQVFQV0Vv?=
 =?utf-8?B?UTU3SFRYMkEyYXowZXN3UnpXNFlhcllMWU1DSXpLcUgzQVNJa1QrMXBkc3Bh?=
 =?utf-8?B?bWdEMzhRYVY0Q2tiVXlVVUorU3h4SDA0YVZTTUoyN3BDMXFUdGZnNnhOa1k0?=
 =?utf-8?B?TWVHelQ3Ukk0YzVIcFB0MitSV3FvQmlJTjB1UnU4R3RNUU8yVW5NYzJWMEVG?=
 =?utf-8?B?aHd6bnludTNndUdMZWtqN3lwVTRqNUZxbFQ1VHBwQVhRTkpyTXlEYjRlYlNO?=
 =?utf-8?B?cVNRM0NmcTJTdlpjcllDNE4zZFNmaDhwdktjMVVzTDdLcklJZnRzWE55RVRz?=
 =?utf-8?B?YW9nRTc5Tm1LNjRwMytNalBCRWtKTzB2VEp0dDF5M1pNemZxaFF0TkxlMGhI?=
 =?utf-8?B?Z080S0xZRWhaTFJSOUdxa1JjaVdYODhSM3ZvZXZldjlhbVZvcHFuMW5LU29B?=
 =?utf-8?B?L1J0b0NDSlBvWElyYlFTSitNK096dVpFZnFtU2NWUmMxdTNGczFwTzZWWjdy?=
 =?utf-8?B?Q1E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e92b0024-d8d1-4680-ce1a-08dbdcbfb9db
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2023 22:53:41.9399
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FDFFbaKA+E2oIIu9ql4DwqrD7d7lOyzigiqe3n1OBR9Ae3fl52KwXtbYvbwu0+CkxX8aBmUQPDmydGVfL5a9hPi1seT8T3ZG4GPz/Qb1hTQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR11MB8723
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-1.3 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 03 Nov 2023 15:54:06 -0700 (PDT)

Hi Ilpo,

On 11/3/2023 3:24 AM, Ilpo Järvinen wrote:
> On Thu, 2 Nov 2023, Reinette Chatre wrote:
>> On 10/24/2023 2:26 AM, Ilpo Järvinen wrote:
>>> L2 CAT test with low number of bits tends to occasionally fail because
>>> of what seems random variation. The margin is quite small to begin with
>>> for <= 2 bits in CBM. At times, the result can even become negative.
>>> While it would be possible to allow negative values for those cases, it
>>> would be more confusing to user.
>>>
>>> Ignore failures from the tests where <= 2 were used to avoid false
>>> negative results.
>>>
>>
>> I think the core message is that 2 or fewer bits should not be used. Instead
>> of running the test and ignoring the results the test should perhaps just not
>> be run.
> 
> I considered that but it often does work so it felt shame to now present
> them when they're successful. Then I just had to decide how to deal with
> the cases where they failed.
> 
> Also, if I make it to not run down to 1 bit, those numbers will never ever 
> be seen by anyone. It doesn't say 2 and 1 bit results don't contain any 
> information to a human reader who is able to do more informed decisions 
> whether something is truly working or not. We could, hypothetically, have 
> a HW issue one day which makes 1-bit L2 mask to misbehave and if the 
> number is never seen by anyone, it's extremely unlikely to be caught 
> easily.
> 
> They are just reliable enough for simple automated threshold currently. 
> Maybe something else than average value would be, it would need to be 
> explored but I suspect also the memory address of the buffer might affect 
> the value, with L3 it definitely should because of how the things work but 
> I don't know if that holds for L2 too. I have earlier tried playing with 
> the buffer addresses with L3 but as I didn't immediately yield positive 
> outcome to guard against outliers, I postponed that investigation (e.g., 
> my alloc pattern might have been too straightforward and didn't provide 
> enough entropy into the buffer start address because I just alloc'ed n x 
> buf_size buffers back-to-back).
> 
> But I don't have very strong opinion on this so if you prefer I just stop 
> at 3 bits, I can change it?
> 

We seem to have different users in mind when thinking about this. I was
considering the users that just run the selftest to get a pass/fail. You
seem to also consider folks using this for validation. I'm ok with keeping
this change to accommodate both.

Reinette