Received-SPF: pass (google.com: domain of linux-kernel+bounces-91703-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Tue, 5 Mar 2024 13:37:14 +0800
From: Feng Tang <feng.tang@intel.com>
To: Dave Hansen <dave.hansen@intel.com>
CC: kernel test robot <yujie.liu@intel.com>, Pawan Gupta
	<pawan.kumar.gupta@linux.intel.com>, <oe-lkp@lists.linux.dev>,
	<lkp@intel.com>, <linux-kernel@vger.kernel.org>, Dave Hansen
	<dave.hansen@linux.intel.com>, <kvm@vger.kernel.org>, <ying.huang@intel.com>,
	<fengwei.yin@intel.com>
Subject: Re: [linus:master] [x86/bugs] 6613d82e61:
 stress-ng.mutex.ops_per_sec -7.9% regression
Message-ID: <ZeavimfXlL4p1L85@feng-clx.sh.intel.com>
References: <202403041300.a7fb1462-yujie.liu@intel.com>
 <a1a2da34-af20-471d-a637-ddd749ce809a@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <a1a2da34-af20-471d-a637-ddd749ce809a@intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?oDVdBQ8ohHS9Xzdtt1L/Fv/5YPVv31wIJiaZYuuNMa4MsayJmYxNHG5i0J8P?=
 =?us-ascii?Q?2eN/8ubVwM2J45Ad02eOBjOELum+dFYXppWPJIn9hkbFh9zDDJ5041tFeSq1?=
 =?us-ascii?Q?9Ot6CA8DdrPqO0PZ1bi+LyEa5ogpEhFPgrGVFyLerXCRuDES1tQt1Fs7UN4L?=
 =?us-ascii?Q?MKffcO1YrUHUcnSxuRpNGSqCBlIm4uEQWteT3eR/1DUb1icfbv66JGWa3ZmF?=
 =?us-ascii?Q?EzoQ0CUe4mN6pZxhje8JZx6BsH8Cdzb9rdStluNNESsna4zul/3b0EMaJ23O?=
 =?us-ascii?Q?zq5QhRL2sds940n2c9gcZrLMdc3tFtTHcv2mcXvKpbysgZFwuGNnxaVZmboV?=
 =?us-ascii?Q?jn1hLaGb7k8OzI1x4aBXNcAl/mxtdGbmVXZxs6cxXFt1WbtZ6UqvYzVISb44?=
 =?us-ascii?Q?L7B/pIr2fIHaEtX/fwmQEdmSIyuxfU+yyPrVFD2QfU1kVDFNh+uYk3NIWbe8?=
 =?us-ascii?Q?CrdQ66T3GCce970BPk/7ihZzzpeCRhIex9rQQUyVniOKO6+WtB1XyfyoAnay?=
 =?us-ascii?Q?B29Q3CwZ/fyV/uDrHPY9Gqd/4wX5kJ7JOjeJZVvP6g6oKapQ2yo6JlxbZhue?=
 =?us-ascii?Q?geC6CrpAJMjto8MGZBOGjA94+3sYv2eEjwUGLjm464+gjRJXaXDr74Z9cR0R?=
 =?us-ascii?Q?uSLNsadReR3d/tbPc1hX3KJFQdVhvDTEthiZKmugfMpAQz88lsYJWN8QPgH4?=
 =?us-ascii?Q?q5OV4Gmi8+Qv39d8mIgJGSiBsOLzumPB/bsymaInd2dmGpsqo7AJn9L8ciN6?=
 =?us-ascii?Q?+ZDgXcr5rMltYuQ+tKslINri0dbwGiDCFWst6sQRJKd+V7d/jZDf/dU3pasg?=
 =?us-ascii?Q?AAIfdpv9gWT4F1p5r4HTIJtZMyRX0l0lrFay/mCDfJ1beRddsZTRNgFS1qGt?=
 =?us-ascii?Q?XeUERtghlOIccDZP0qQNLELaB8Iv01g4BGjS6qtIbf7dqM3YiDBRPxK8qvpx?=
 =?us-ascii?Q?3QD4KEnH9qKPsxNkg4NF2xF7lzhZdd3zCVA+iZFZJmazctNjVsDuHHPgBAaY?=
 =?us-ascii?Q?+hoR5RbB8ELVaRWkr6CNHJGkE6NweEp1fkwcLOiykALIYzJguJxLqXcyh6Vn?=
 =?us-ascii?Q?Nbz/k6f4+H+TNZKNu3Jb1/sm++/5CYHFlKpv9oVwXqcJOm8LK+OABMSc9NLd?=
 =?us-ascii?Q?BUDxfO8hhNaSbdhy6L1vh+Vc6uA4c3y7xaDdIrxrPK3bhnz3rJcRu+i8RoLb?=
 =?us-ascii?Q?F9N8IwSUgXdIXlvKeXiOjZw/KjL0NPxTKt2R+5RFJWl2lASZc6jAcbX9xOu+?=
 =?us-ascii?Q?nZd6zptuvbaieCKGQX+pkUk2dvSHUCkRBipsBVQac5ltG7FM+GXl9LM/HB4z?=
 =?us-ascii?Q?38Q8hkfYaqpEX32onUhU5sN4LZM3QSYYu4P11CXmTkn4PY75j6/J6n0CmWM/?=
 =?us-ascii?Q?ys0rjrDv20OIxlRHiJzHPklV68tHALKX+g3wpiSoFXN3JzR4NquuOnjRWfv/?=
 =?us-ascii?Q?ZttMMJtE+qAML+egzllLmHFf05T9ogAAQXY3MEJHfl/1SK9NffHcuS51NxBw?=
 =?us-ascii?Q?3MVtaosl7aN66rU0jFNHLaEyJz4T2/0eFE7QGzR0sDRNMg1mnqi8Wbxtbt6u?=
 =?us-ascii?Q?sZ/d5/WY5E2RzorGFqFWc1Lr/0DtftK+KtIzAv7j?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1d9c4c5f-983d-4368-15e2-08dc3cd845ef
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Mar 2024 05:51:16.4568
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: cO0SrXd9s3XNctqYc8NAHZs9bEF8KAIkxQVXFZjo2YotsyFKjhf/FD+DKNCpoMJOk6oNKPOIzlA+NMWneZrRWQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7287
X-OriginatorOrg: intel.com

Hi Dave,

On Mon, Mar 04, 2024 at 09:58:53AM -0800, Dave Hansen wrote:
> On 3/3/24 21:53, kernel test robot wrote:
> > kernel test robot noticed a -7.9% regression of stress-ng.mutex.ops_per_sec on:
> > 
> > commit: 6613d82e617dd7eb8b0c40b2fe3acea655b1d611 ("x86/bugs: Use ALTERNATIVE() instead of mds_user_clear static key")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> 
> This _looks_ like noise to me.
> 
> Some benchmarks went up, some went down.  The differential profile shows
> random gunk that basically amounts to "my computer is slow" because it's
> mostly things that change when the result changes, like:
> 
> >     182670            +9.0%     199032        stress-ng.mutex.nanosecs_per_mutex
> 
> Does anyone think there's something substantial to chase after here?

We further checked this, and it seems to be another case of data/text
alignment effect, that 6613d82e617d removes staic key 'mds_user_clear'
which sits in '.bss' section and change the address alignment of
following data in that section.

With below debug patch to restore the alignment, we can see the
performance is recovered:

   a0e2dab44d22b913 6613d82e617dd7eb8b0c40b2fe3 398e7f0da8595354dc330938831 
   ---------------- --------------------------- --------------------------- 

    302318            -7.9%     278364            +0.3%     303161        stress-ng.mutex.ops_per_sec

---
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 48d049cd74e7..1876865dc954 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -111,6 +111,9 @@ DEFINE_STATIC_KEY_FALSE(switch_mm_cond_ibpb);
 /* Control unconditional IBPB in switch_mm() */
 DEFINE_STATIC_KEY_FALSE(switch_mm_always_ibpb);
 
+DEFINE_STATIC_KEY_FALSE(test_static_key);
+EXPORT_SYMBOL_GPL(test_static_key);
+
 /* Control MDS CPU buffer clear before idling (halt, mwait) */
 DEFINE_STATIC_KEY_FALSE(mds_idle_clear);
 EXPORT_SYMBOL_GPL(mds_idle_clear);
---

There was another similar case which changed the alignment of
percpu section: 
https://lore.kernel.org/lkml/ZSeF6T0mkrH5pOgD@feng-clx/

Thanks,
Feng