Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
Message-ID: <cf7439c4-f72c-a145-5a65-84ae15c5d96f@intel.com>
Date:   Tue, 12 Sep 2023 15:10:53 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.15.0
Subject: Re: [PATCH 5/5] selftests/resctrl: Reduce failures due to outliers in
 MBA/MBM tests
Content-Language: en-US
To:     =?UTF-8?Q?Ilpo_J=c3=a4rvinen?= <ilpo.jarvinen@linux.intel.com>,
        Shuah Khan <shuah@kernel.org>,
        Shuah Khan <skhan@linuxfoundation.org>,
        <linux-kselftest@vger.kernel.org>,
        =?UTF-8?Q?Maciej_Wiecz=c3=b3r-Retman?= 
        <maciej.wieczor-retman@intel.com>
CC:     <linux-kernel@vger.kernel.org>,
        Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>,
        <stable@vger.kernel.org>
References: <20230911111930.16088-1-ilpo.jarvinen@linux.intel.com>
 <20230911111930.16088-6-ilpo.jarvinen@linux.intel.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20230911111930.16088-6-ilpo.jarvinen@linux.intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VU9Hdis1Uzh1N2l1Vlg0TWRNMS9zSW9walhod0VLNGhQcm9qNHJvQ1RXbkRY?=
 =?utf-8?B?LzIzdERqMGNaNnNVYm5KVkFiV3dwYU5zemI3NURlMHVkZnJwamdjc0NSZHVH?=
 =?utf-8?B?QmM5bVZGRjNaaUJDc2w2M3dLbHNiTHV2cVR0RlhRckU1SWg5Wm1sMlNTbjZW?=
 =?utf-8?B?ZjB5SFZianpORlBmenZIZFlaZTIrSU9iV204N3EwUlB0ZUp6S2J6OC9uMVhS?=
 =?utf-8?B?UGErNTBtVzg4aG0yNUJGV0RDVWQ4MHJ3M3pWMGhlNFEyZ2NGWGZNQnl0bXZt?=
 =?utf-8?B?VXRMbFczMTh2b3pRODI3eTRyNC8xTkZJMkJVdjNLMnVwRHlIS20zS1lJTHcv?=
 =?utf-8?B?VTVHVnloeEhUNnZuRXlGYnA2NFVsaVdmaTR2a0YvWlhNZ1ZWRGNEMUdVUmwy?=
 =?utf-8?B?b3N5eXlZWU4xeUluNkNvVFREMXJMZ1lFQlE0a3NtazYySzdIaTdReE5yQlIz?=
 =?utf-8?B?b0dZQTR2QUlDK0E4UlBvTFRZUWtmQ0lMYjc5MTRmcWN4Zm52bjNNdVduZmUv?=
 =?utf-8?B?b2tWQlFzdHIxL3hDR08ySWdHWjdYcHpvZWQxZzNIV1NDRSswOTNOZURGV2hT?=
 =?utf-8?B?akdWUlREM3RNVzdwZEFSWkxZYVFPa0ZPUlEzS0dlL2Z1NDVlTys3OHpRNUlY?=
 =?utf-8?B?NnFFa2xBeU1GQ3VOemtqSEdhTEUrdjZ5WFE2TkNOK2lHSFYwRlEySXhLV0RT?=
 =?utf-8?B?bm5KYkNxR0tXYXVGdFdRL1NjS1Z5S251L3lDY2orandiWE9WSVhhNUhndkR6?=
 =?utf-8?B?dXV5MnF2L1FDZENlL0wweE5ndkEzc0ZxRVY5T0dYRGxkL1BHTGszSS9KWnp6?=
 =?utf-8?B?OFRPYXhSY0JQdXVDUngwR2dSZEJhdWNab2Q5Yng3RlJwSndTamhxYklMNVkr?=
 =?utf-8?B?b3l6ZWJjbUlwWjUzRTNMajdPUTB6OGVFS2N1ejdBWlRMSGdLOG5xRkpia3Fo?=
 =?utf-8?B?bFdtd3BPdVFjVTJsdC9qbS94enU5bC8waEJBTE90MFBhWFJNZyswdG93aGhE?=
 =?utf-8?B?Sk1oUnBDQXYyTEoyUjZIVFV2QUp1Nkk2UjRqTkc1THZIRmJPNDlkS3h4bE4r?=
 =?utf-8?B?M1owQVFUNjJJdTdVakdZWFJQSFFwYWU0RVAwemFjSzI4ditIYXZITmtMaE1r?=
 =?utf-8?B?VG9pKyt3ZXo4VUdrQWdNandHWlBWWUtlZEdZbysvNmQ2azh6SVR0NFAyY05G?=
 =?utf-8?B?bXFGOHpWYWFDQmNWZU9keUFPTWVvOXJiSWZhcUtIclZ3SkxqbFBwMXRqZzIv?=
 =?utf-8?B?SlBOdW4vNFM5TkpZVWRBTlc1a3FLUWxYdDJkeTFEbTNSL0VyQlliOTRZL015?=
 =?utf-8?B?MEp2VHJkYWJwWWloSE1HbGlGQ291T1NqNHd3U3FLV0huSm16V3lYYmJBWWpp?=
 =?utf-8?B?S0NlVkJQVlZNTmovM3NpOVRvWTRvWHJHN1hhREY5YjkrVlVBMTl1M0JSV2VI?=
 =?utf-8?B?MWZuaVJZVlA3ZDRydlNTaUVTWXQ4UUhiS1QzM2V1M1ZOY2szOVNXTFRZMWlH?=
 =?utf-8?B?VkxLcXVKaGlxSmhjckhYQkRvMzJ6NHJOVUZsNWphdlVQRlpGNnNkeWMxU2x2?=
 =?utf-8?B?WERlcFVBWlFmSFVzWEVrOVNiMVZWNXRNdHNxUmlWOEVlVklEazBsLzlZSWFN?=
 =?utf-8?B?VFcvMU0yKzlwRWtyQjBBdnF3UElTSVc2dGpVeUFlQjEycDIvdnhLVFZpY2VP?=
 =?utf-8?B?aU9iSTJjUlpHRkxDVFZWVyszZGxzMkJ5MGhUR2FJWFY2RnZ4MDBLY0VLbGp3?=
 =?utf-8?B?bXo4N2RIaWZhVjdqQkN1KzNGdmM4UkQ2Z1VIdlRZZlRGblNaL3FKWE9nMC80?=
 =?utf-8?B?TzBZbFczd2FiSjh4ZUpveXQ2VjFnTXlCOENKbTVDc1diUnF2c2pmMHFsVVY1?=
 =?utf-8?B?YjVHbW5lRFd4UWlvYmRhdXRHZDhnWkdtSTVuMHQ5aWp5SllVL0pzbzJKUG9I?=
 =?utf-8?B?b292N3NmSWpwVmt3V1p5cjhiSlR2RWdTZ05Nc1QrRjd1VzFFc24yV1NVNEdU?=
 =?utf-8?B?YjdYQVE2Rkp5ZFM4SkpBVGRnZ0E2dGVuMkNRY3o3RnBFamtPamp5c1k3VlpF?=
 =?utf-8?B?UnB4NkRvOU9FanVEcnE0azBrS1ltb2k3bTdlSVQzaWJlSCtDcDR4WTV2NE1O?=
 =?utf-8?B?V2lmcll1eDMxYmhmL01HNmEyRXAwUklkNnVSNjdUWWhBNEo0RXFkOXVRdFh6?=
 =?utf-8?B?M1E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: df7db4d3-d090-4567-1c2e-08dbb3dd229c
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Sep 2023 22:10:55.3817
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: zliYG7JFMvcpOxtVFBX1FXIvZPRhKPDEm3YzHyKSh6Al4u2z6x1MUGtsCQ34TfmxRStwdSvI0wwY1mcM5o+M1wHgXDg530Ww9AdPUeMmHOQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB6029
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Tue, 12 Sep 2023 15:11:19 -0700 (PDT)
X-Spam-Status: No, score=-2.3 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email

Hi Ilpo,

On 9/11/2023 4:19 AM, Ilpo Järvinen wrote:
> 5% difference upper bound for success is a bit on the low side for the

"a bit on the low side" is very vague.

> MBA and MBM tests. Some platforms produce outliers that are slightly
> above that, typically 6-7%.
> 
> Relaxing the MBA/MBM success bound to 8% removes most of the failures
> due those frequent outliers.

This description needs more context on what issue is being solved here.
What does the % difference represent? How was new percentage determined?

Did you investigate why there are differences between platforms? From
what I understand these tests measure memory bandwidth using perf and
resctrl and then compare the difference. Are there interesting things
about the platforms on which the difference is higher than 5%? Could
those be systems with multiple sockets (and thus multiple PMUs that need
to be setup, reset, and read)? Can the reading of the counters be improved
instead of relaxing the success criteria? A quick comparison between
get_mem_bw_imc() and get_mem_bw_resctrl() makes me think that a difference
is not surprising ... note how the PMU counters are started and reset
(potentially on multiple sockets) at every iteration while the resctrl
counters keep rolling and new values are just subtracted from previous.

Reinette