Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7;
Message-ID: <37888890-bfba-c3d9-558f-4c6e830669f0@intel.com>
Date:   Tue, 26 Sep 2023 14:41:59 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.15.1
Subject: Re: [PATCH v2 6/6] selftests/resctrl: Reduce failures due to outliers
 in MBA/MBM tests
Content-Language: en-US
To:     =?UTF-8?Q?Ilpo_J=c3=a4rvinen?= <ilpo.jarvinen@linux.intel.com>,
        Shuah Khan <shuah@kernel.org>,
        Shuah Khan <skhan@linuxfoundation.org>,
        <linux-kselftest@vger.kernel.org>,
        =?UTF-8?Q?Maciej_Wiecz=c3=b3r-Retman?= 
        <maciej.wieczor-retman@intel.com>
CC:     LKML <linux-kernel@vger.kernel.org>,
        Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>,
        <stable@vger.kernel.org>
References: <20230915154438.82931-1-ilpo.jarvinen@linux.intel.com>
 <20230915154438.82931-7-ilpo.jarvinen@linux.intel.com>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20230915154438.82931-7-ilpo.jarvinen@linux.intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZktSQkdxblM1ZGNDNTR6UFBSR3BJbXNRYll3QkFSUnp4UDhWQ081Q0IrT0hN?=
 =?utf-8?B?alZTcDlmNVBVYmwyLzg3cnZpS2U0Y3dKZ2NoblEyZ2N1bmExNm50emFoWHh4?=
 =?utf-8?B?MlVVa1l3SWdUWmdkVkk2czU5aVJnSXhxZzhqQm9URFVpdFRCdmRVR0JEWUU5?=
 =?utf-8?B?akc0cDhickJiL0d3MFVVdGhVN2JwRXg0eHpzemFrbjl5UHFuQ0RvV1ZjYmlB?=
 =?utf-8?B?cG1xRDJmcXNkTmhyTWl3SkpwbFlhSFBxbVZOZG9wbFR5ZGVOcCt1L2hOOXFQ?=
 =?utf-8?B?aGcybnRwZHUvZFdIVkg3OHZYMHgxY1BTanUzUE5rSlp1UHdMZ2h2aXJiblJs?=
 =?utf-8?B?WFBXWGEwQkFtcTY3bXg0ZTh0RWtyK1NoZW4zb1gyNHRCdVJsQUdCZHJnWGgv?=
 =?utf-8?B?c1d1K1ZRVkJRcDZzWkVDN3FQY2djOExFbGtDbnNVUmQvbVRpYy9DUGVZQndJ?=
 =?utf-8?B?aTMwL0tCekNGNy9hVVR4d0o5S0l2MlFlQXFvUVVFTGo4SHZ0ZUFCQnFtZTlF?=
 =?utf-8?B?K0drYlhpQ1hoUjV2VnJES2FzODlEU1kvZlU0bWNDWmVWK3JBV1hpQTc1UlNH?=
 =?utf-8?B?bHJVUU1oMzJqOGtsNm4zQzVlZTllMEFhQW9hM2k1Z2lJaXVvTjFpd0tJelVy?=
 =?utf-8?B?R3QzSUYvcEJFVFhwbEE4a0M1MGJvcEd6RTFlSXpyaFBNVmNRK1ZrOUpkVDV6?=
 =?utf-8?B?eWlPbzBCRGhQNlJoSVNQd1d4Z1RTd0FEa1F1UXZKTE5JcXViTDhtVDNnOVBD?=
 =?utf-8?B?SmpHTWNHRkI3RlFMalQwSlZQaVhmRWZmUnJyT2VVcEJiUkV6TUtEcmFSYStJ?=
 =?utf-8?B?V1lKZVRnYmRSQ2lzWHo3SndyRVJlR2RQcklnbFVhV0doTlhjRWRUalRyM3hY?=
 =?utf-8?B?QWFtZWlGSUI3Z3g5dnRPYW9vdnNVSlhrT0xUVmpMbGpnMWRhTTZUTlhxNFpD?=
 =?utf-8?B?NmZINWluR3FjMEJZU0VTajliQXNiQUsyYVN5YStKMk1rS3B3aFFaYm9TektR?=
 =?utf-8?B?Qk9JUkNiSnZvYTRIZUFFMWZYUUdGOTdBZFArNnU5ZFBDRkg5Z1JTWlpIbTBF?=
 =?utf-8?B?RjdsZ01leHRUMlBKemdxYlpvV3dpejBucDlrVEY0TE1SU3BEV2VTZStucnQw?=
 =?utf-8?B?Z2RGR0JjWWo2NGhycWdrUlpnVzdrbExVc0JjOGFlZ0o2OFlMRnp4R3piWkpQ?=
 =?utf-8?B?T09Id0xLOFFUNW03aks4Wi9WQ21hNkc4WTVnZURGcmdxMDJVQzZqa3I2OUIr?=
 =?utf-8?B?RWJHYVQzZVJCbVQzTUJkdDJEUlRLWGlrOGFrRmJJd2RUMWRBeURKcDMrUmZT?=
 =?utf-8?B?YjRuT1dhNnpGV3RUenRUL1BOL2hjZHNiTHQ5SXdwVmY4c01pNkNDYVFSbVlj?=
 =?utf-8?B?dnNLVGlJRVF4dzJVRGhHSGg2R1JXR25MdEFrWGdWOS9pejgvbExxRERpZlIw?=
 =?utf-8?B?N3RBOXZuR1dwRGdSa0ZaQ1ozQTAvV0JmNjdRS1BwTzlpWWhtcW1KVnFaUS9J?=
 =?utf-8?B?bnBJREtxSzN5eFRvRHYvc080VnpaNWkrNmRHQlViS3BNSUU4VHpwZ0lkckFR?=
 =?utf-8?B?M1kvVWJMZDRlNHZaYnZ5ODdqTnRHcG1xV0tjNVdQRWJ4bmFWYWJrY2dYNE1N?=
 =?utf-8?B?amtYNGFQUWY4dndsNjkySzVkd2p1K1BlM1dNVDJ1V0hXSGVla1RmcTZkUnlZ?=
 =?utf-8?B?STRaNC9McGRjK3JrNXlXRTJ4SXBOdG1wSGF5dmd6VXk2UDhkMkdHY3IwUkc4?=
 =?utf-8?B?WGZmUG9RbVNBZ09QekFIRWkrV3NUZGMxNHJQM3pjbDVXOFY2cUhaTFB2TEZu?=
 =?utf-8?B?UnNNWEQvcE5RaW11bDN0R0svWVF5Z01FVGdOaWxzTHRsV0lXclRSS202UGxJ?=
 =?utf-8?B?VVAydnNRWDVBUmJic2M3RzQvUEw1NWlPcXZoalV5c2IzcFRDcEhzeVQ0aEZo?=
 =?utf-8?B?ZlNLL0Jpd0pMZDRwdkl1Q3FuQzdoSEVHdDVETld4M1lYOW1DWjlUZkN3VHFD?=
 =?utf-8?B?WFpDWmhrK2pBRjNEK3NHY0E1S21XS1loRUtweWhESEkrTVVFVDg5akJEREJX?=
 =?utf-8?B?Nnh0QUpESloyNm5aZUZSUFZkL3pmZ2VMR28zcVpmYjcxdmQxaW0wcjNadTB4?=
 =?utf-8?B?TFE4UFJXRFRPSnI0TnoyTEEvbXlXSzRUMFFNbkNnZmRzMFRiZXpDdGJJY2pz?=
 =?utf-8?B?dHc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: fc10650f-455c-495b-bf9a-08dbbed96b83
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Sep 2023 21:42:02.5035
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Ck8lTLdwDjfdW8kfnM6VMtpdTwiHzSGH2kyxkKbKWA9hhdEEURs1Fx7uMcw2T3+6MIXubR/IVnnEncDx7C+ue6pLEn5a5rXtGQkd9fR2NXk=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR11MB6276
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.9 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,
        SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Tue, 26 Sep 2023 16:42:51 -0700 (PDT)

Hi Ilpo,

On 9/15/2023 8:44 AM, Ilpo Järvinen wrote:
> The initial value of 5% chosen for the maximum allowed percentage
> difference between resctrl mbm value and IMC mbm value in commit
> 06bd03a57f8c ("selftests/resctrl: Fix MBA/MBM results reporting
> format") was "randomly chosen value" (as admitted by the changelog).
> 
> When running tests in our lab across a large number platforms, 5%
> difference upper bound for success seems a bit on the low side for the
> MBA and MBM tests. Some platforms produce outliers that are slightly
> above that, typically 6-7%, which leads MBA/MBM test frequently
> failing.
> 
> Replace the "randomly chosen value" with a success bound that is based
> on those measurements across large number of platforms by relaxing the
> MBA/MBM success bound to 8%. The relaxed bound removes the failures due
> the frequent outliers.
> 
> Fixes: 06bd03a57f8c ("selftests/resctrl: Fix MBA/MBM results reporting format")
> Cc: <stable@vger.kernel.org>
> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>

Thank you.

Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>

Reinette