Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7;
Message-ID: <ae5ab387-0a1c-3613-5c04-db879aebc856@intel.com>
Date:   Wed, 13 Sep 2023 14:00:19 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.15.0
Subject: Re: [PATCH 5/5] selftests/resctrl: Reduce failures due to outliers in
 MBA/MBM tests
Content-Language: en-US
To:     =?UTF-8?Q?Ilpo_J=c3=a4rvinen?= <ilpo.jarvinen@linux.intel.com>
CC:     Shuah Khan <shuah@kernel.org>,
        Shuah Khan <skhan@linuxfoundation.org>,
        <linux-kselftest@vger.kernel.org>,
        =?UTF-8?Q?Maciej_Wiecz=c3=b3r-Retman?= 
        <maciej.wieczor-retman@intel.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>,
        <stable@vger.kernel.org>
References: <20230911111930.16088-1-ilpo.jarvinen@linux.intel.com>
 <20230911111930.16088-6-ilpo.jarvinen@linux.intel.com>
 <cf7439c4-f72c-a145-5a65-84ae15c5d96f@intel.com>
 <c1518af-cc3c-3aa7-a3c-4bbfe8cc6cd@linux.intel.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <c1518af-cc3c-3aa7-a3c-4bbfe8cc6cd@linux.intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NHlMc3JkcWx6T2ZQN3JsY1IxNzhURTZxaHpFeVh3R3JWZXlSaUw0bkloRklU?=
 =?utf-8?B?OGtteEhPVG9UYzNSd3ZVVTA3d3NObDNMWmxJVjlUdHFSL2JOU09pa3pQcXli?=
 =?utf-8?B?akk2V2ZybGZRei90TXAvQ0hNYTRsdnExT3RDNnlCNlNWYkU5R1U5WU85b1Nu?=
 =?utf-8?B?UVJlMG5oUm5EakpBdmREREN6dzUxWENJbHh5V0RXZU1Yd3lxRTZOSENYZ0FP?=
 =?utf-8?B?ZkdSN2tRWVRKRVByM1JoZzVOY1dqc1hDSHNQdGhnc2JsbFgweER1elNOUkVj?=
 =?utf-8?B?UktjLzhTazRyUXZXaU1vNytiVnJLTldPMDlQUFJIbGx3VGxIdVdMR2xVOUpR?=
 =?utf-8?B?SjRWM2ZGcHdGK1JWbGdJdU5PMFBSOWd0OUsvcnZWSjFYSTNMUG1SUFF1MDdm?=
 =?utf-8?B?dERJdEJ4bGVoZ3NURmlHS2pDMGRVdlh6S1o2LzVRY3hHczVnaC9zZkhLYnUw?=
 =?utf-8?B?WEF6amgrMVNYeEROYzI2cG5xZzBSaVhYTlo5dFI5Q3dwSzRWN1FySnJmbUJt?=
 =?utf-8?B?em9HNTczZ2VPVFFvZjNLWklEOWRzKzZHWDNoM1hnaEI2a2U4V3pCS3lOKzYy?=
 =?utf-8?B?NnBkVXRVM0VmcWpZd2V6Q2duUHkrQXZZYk00dmd1TXVhOE9IRm11NzlWME5r?=
 =?utf-8?B?ajRadHBSTGxLNko5RTVXcWR5RzBscGsxRHBmSVIzQno1Y2NaUU00WU5tb0ls?=
 =?utf-8?B?TFkrYXBwM0FPZmlJdDB3RTF6TzFQUmR6L0syZXExR2YySzNCNGFOa1JPaEFs?=
 =?utf-8?B?R2xPdFUreWdDd0QzRmFCdW1xdmNId2tUSDFVN0ZYZndSVXBKTWp6SjRGRXph?=
 =?utf-8?B?VmRHVnRSczRBZElEQThhM0QvWHZ2QmorMkgybTFqVFZhdTk5RWlaZ1l0Y3pG?=
 =?utf-8?B?cC9IWVdjS2EvMmlUZmlVdm5xNzlIWWp6NmFmeXJueG94SVY1MlhJZk5wRmp2?=
 =?utf-8?B?NFNXQU9BYmdYQ2hSdFVCSmxtNDZGaVZwYllPaE1iOEJIQ25ibXpHbG9sN1pq?=
 =?utf-8?B?eS9JWldXMHBFZXFHWWdiQnFUTU1nMEdVVTVIci81Y1lkRzhGYzY4S0Q3RFBk?=
 =?utf-8?B?dEVSazhWWEg3NXgzOUhzY1gwOVk3cVFsMldIQ2dSZkE5b2hXM1pFdHk2L3Fo?=
 =?utf-8?B?UkZ6OU12Mkp3dzRKOGRNa2NPQ0licUNSd1ozc2orSVZTbWJwVXVnUFQxcDdk?=
 =?utf-8?B?cnlhb1JFbVNTaUdLTDA5ejF2YmEzcnJBb1FqWlBqNlU2TUM4VXA4bGNlMzAv?=
 =?utf-8?B?a1hTclBuWXN3R0plTkEwbW8xVkhpakdMRzU5RVRWdXNwSDRKamo2Y2I3OXlC?=
 =?utf-8?B?V0orWmJMYmVQb3BQOURDRnJYZE5HR1c0SDVUWVVsUklCL2x1N05ocUxaMnJv?=
 =?utf-8?B?a2dQZGwyZE9rL2I5Z0g3ZGF3NnZGNElwNVlOWUQydENsMTE2Rm1FNHJpQzdp?=
 =?utf-8?B?czJqeVVvNjNlYnNiUUdpRjc2Y1hBTzI1SVpBMk5VM2hQV2V5dmtpTlZ6cy9B?=
 =?utf-8?B?RTFNNW9zaGRMcXFYeDFqNDB1VU1OWXAxOGdjMTl0R0JLcys3c09hWm5qc21p?=
 =?utf-8?B?UkZ4bGlBempPalUrb3cySVJPc3FaMytBczZDeE9HaWxUTWV5aFNPS2lGUVI3?=
 =?utf-8?B?UG1Yd1ZucHhQS0d6UEZBQWJ0Wks2blZUcW55RXBNZUFjMVM0YUwycVA2bXd1?=
 =?utf-8?B?a1J1Zm9pVEFCQUJxdmRpZHhaczNBdUR2NjZsVnZScXpCQW5Ua2swdU52M2tn?=
 =?utf-8?B?cnFOT2xZZDZIc2oyOFpTMUdvQVdEOEFZNFpZRUxZWmY4TDRtSWdaY2s2YVQ0?=
 =?utf-8?B?c0djWWpSMHdydDNqbE5NVDU4TmZyN2MwdndNREtKSks5dEpqemN0WjRYdUhL?=
 =?utf-8?B?bVVJS29qUnV2SU44ekNvY0FSMkdWZU9DKzhBU3Y4d3NMbDVRS0dDRXZOUFFO?=
 =?utf-8?B?V0JxMkpreVNadE9iYVdxVk5zc3BJU0dLSk12UFZKekdKN2NLek9MRERqTXhV?=
 =?utf-8?B?OFp4dWZuTDYyTk94Mnk0eC9hS1o1cmNuRG51dE1oclRXMVZmL012cGl4eDNs?=
 =?utf-8?B?WUoxVS9TTGNkVWN0YTB6VEFkZzVDQmUvYy84SktZeXJJRjByT2VBUUpIY0Yv?=
 =?utf-8?B?eXcvMElKZXQwM3VNUW82aGtGeDh4R2Mvbk5STnBsMnVSOFpLZitYZDREYnhp?=
 =?utf-8?B?L0E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 2569358b-6f7e-495b-3dbb-08dbb49c7192
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Sep 2023 21:00:21.7863
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: MeyWBlog1g6F7XKlpBqjIhHvgfykWegF2SF/3aNXTBa/sU0ZljxcOSWBeepRCl+GJABLR37pNtPMfOc7dl34AOnKsJXkh5nfEzGBWN73l48=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7811
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 13 Sep 2023 14:01:45 -0700 (PDT)

Hi Ilpo,

On 9/13/2023 4:43 AM, Ilpo Järvinen wrote:
> On Tue, 12 Sep 2023, Reinette Chatre wrote:
>> On 9/11/2023 4:19 AM, Ilpo Järvinen wrote:
>>> 5% difference upper bound for success is a bit on the low side for the
>>
>> "a bit on the low side" is very vague.
> 
> The commit that introduced that 5% bound plainly admitted it's "randomly 
> chosen value". At least that wasn't vague, I guess. :-)
> 
> So what I'm trying to do here is to have "randomly chosen value" replaced 
> with a value that seems to work well enough based on measurements on 
> a large set of platforms.

Already a better motivation for this change. Your cover letter also hints
at this but this description does not mention that this is not just
another number pulled from the air but indeed one that is based on
significant testing on a large variety of systems. This description can
surely mention all the work you did that ended with proposing this new
number, no?

> 
> Personally, I don't care much about this, I can just ignore the failures 
> due to outliers (and also reports about failing MBA/MBM test if somebody 
> ever sends one to me), but if I'd be one running automated tests it would 
> be annoying to have a problem like this unaddressed.

In no way was I suggesting that this should not be addressed.

> 
>>> MBA and MBM tests. Some platforms produce outliers that are slightly
>>> above that, typically 6-7%.
>>>
>>> Relaxing the MBA/MBM success bound to 8% removes most of the failures
>>> due those frequent outliers.
>>
>> This description needs more context on what issue is being solved here.
>> What does the % difference represent? How was new percentage determined?
>>
>> Did you investigate why there are differences between platforms? From
>> what I understand these tests measure memory bandwidth using perf and
>> resctrl and then compare the difference. Are there interesting things 
>> about the platforms on which the difference is higher than 5%?
> 
> Not really I think. The number just isn't that stable to always remain 
> below 5% (even if it usually does).
> 
> Only systematic thing I've come across is that if I play with the read 
> pattern for defeating the hw prefetcher (you've seen a patch earlier and 
> it will be among the series I'll send after this one), it has an impact 
> which looks more systematic across all MBM/MBA tests. But it's not what 
> I'm trying now address with this patch.
> 
>> Could
>> those be systems with multiple sockets (and thus multiple PMUs that need
>> to be setup, reset, and read)? Can the reading of the counters be improved
>> instead of relaxing the success criteria? A quick comparison between
>> get_mem_bw_imc() and get_mem_bw_resctrl() makes me think that a difference
>> is not surprising ... note how the PMU counters are started and reset
>> (potentially on multiple sockets) at every iteration while the resctrl
>> counters keep rolling and new values are just subtracted from previous.
> 
> Perhaps, I can try to look into it (add to my todo list so I won't 
> forget). But in the meantime, this new value is picked using a criteria 
> that looks better than "randomly chosen value". If I ever manage to 
> address the outliers, the bound could be lowered again.
> 
> I'll update the changelog to explain things better.
> 
> 
ok, thank you.

Reinette