Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35;
Date:   Thu, 12 Oct 2023 13:56:06 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     "Luck, Tony" <tony.luck@intel.com>
CC:     "Sang, Oliver" <oliver.sang@intel.com>,
        "oe-lkp@lists.linux.dev" <oe-lkp@lists.linux.dev>,
        lkp <lkp@intel.com>,
        "linux-edac@vger.kernel.org" <linux-edac@vger.kernel.org>,
        "Huang, Ying" <ying.huang@intel.com>,
        "Yin, Fengwei" <fengwei.yin@intel.com>,
        Borislav Petkov <bp@alien8.de>,
        "Yazen Ghannam" <yazen.ghannam@amd.com>,
        "Smita.KoralahalliChannabasappa@amd.com" 
        <Smita.KoralahalliChannabasappa@amd.com>,
        "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
        "x86@kernel.org" <x86@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "patches@lists.linux.dev" <patches@lists.linux.dev>,
        Linus Torvalds <torvalds@linux-foundation.org>
Subject: Re: [PATCH v9 2/3] x86/mce: Add per-bank CMCI storm mitigation
Message-ID: <ZSeKduDig1Z7ZuGN@feng-clx>
References: <20231004183623.17067-3-tony.luck@intel.com>
 <202310111637.dee70328-oliver.sang@intel.com>
 <SJ1PR11MB6083D191286779302821A7EAFCCCA@SJ1PR11MB6083.namprd11.prod.outlook.com>
 <ZSbCYt35j20ezT98@feng-clx>
 <SJ1PR11MB6083154D148B42B8B7BC48D9FCCCA@SJ1PR11MB6083.namprd11.prod.outlook.com>
 <ZSeF6T0mkrH5pOgD@feng-clx>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <ZSeF6T0mkrH5pOgD@feng-clx>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?K5nQ3Rj/Hgin1/p+Kg5COZoivLC9ANyyeJggcic4FJgm+EYNDaoDBqE7lPhK?=
 =?us-ascii?Q?LsEtVv96f2l4NUHlqWVm0IRhkn+IBBRpzB4p7kTDnXubl9gtzwQ/dIfJqiLE?=
 =?us-ascii?Q?cbhyVTqx9Gvtzv4TdFvz4ZzRTB537gxZUpsm6Aw8UClJBM0jFcfmc6uW/liN?=
 =?us-ascii?Q?G6N1Yh7/3w9O7Xy4wJpV5ObXRpIQjYIcvKt7b8SwfKLqmqxcFl1XjMyQrOIM?=
 =?us-ascii?Q?nbLPJN93S8Cr2AXuLUNluBrXAJbngMs/uZvNq+NJcowqoCAHhl5RIFhgKJXG?=
 =?us-ascii?Q?iASj/D4Fsha6pjYWaAosAD5kPLW9fDG3yABdpLT0najyOOD8bWGT7RrcTA1Z?=
 =?us-ascii?Q?J7H5AwVDzdYdRKmE+0TvDmCuo9Xu6XQKoA6Ta2sF3d6SPoWD50KrgXkjM/9H?=
 =?us-ascii?Q?8Vz801FnTCLLuD36XK0A7emyXt0kHhxsk/eNnyorZJ6mQYv5D0qPJ7WUfZBL?=
 =?us-ascii?Q?E3Fg9Bo89SG4vxWA+7C0HUIGrdYL/g6E8OGq4uu/vQdF8Hpf/ImcSAAS4q2s?=
 =?us-ascii?Q?+bgcekqkag1a9eJuDeSpAFGNZ/4nbGFfODHY7FAb0OOWgvV94AgzPrudaX+V?=
 =?us-ascii?Q?i1cbPJc2s9sovfmLwykM01IzyxsSzUW88lEjimpCFdYHl19R9yLzba7/KYZW?=
 =?us-ascii?Q?QqFqD6/WBfbVfQI7PsT6rJ7NE7ouT8abwA8xcY2vT4/pc/ilwGhwUosbQsX4?=
 =?us-ascii?Q?FgsOUTxpv8PAR8PnN6ug+EGkpPL2CimlsfWt6zgGA2feFPSHr4tNTEgJwL+N?=
 =?us-ascii?Q?9dNCOb0UqYwGpQz5xilhwJBhJ7EKfJmlGboEh9EW/4Oqw6UNyAr7Bl4i+Ntv?=
 =?us-ascii?Q?jmGSyTyyr1VVwdlaI5U3TF5wYLdj8ALQ5V5yh+URhfAKWQORIVgcSSRf0Ane?=
 =?us-ascii?Q?cuBbCkaFKS04eNUBHrvmvptQfWKazAfiVUT/Vett3/cI7nFfozynaIfWxJx0?=
 =?us-ascii?Q?4Ou+heKSZJuqlGBUdYCcWnDJ9pHLkOJjRUW5jfhkvSeZ/rE/HmKHKSXAYud9?=
 =?us-ascii?Q?1AB1cU3R4vu6tLd4o8/p6CZniZhsQnW3Fsss8NJkQJF4WGVjTsA2IyNXVETs?=
 =?us-ascii?Q?AAK15DTJ4s0d2HNdh7g000XsRd9jwRVRtymdDAEpfXBTYBORsX9volV6PXQu?=
 =?us-ascii?Q?6vSUIIAwpSiJj6toJ4AiwFTnWgGrl2okPlcgwvm2h6EaWIHEkTM6ZrjpN8fv?=
 =?us-ascii?Q?GHUzy8XZ9u/QYws1v9e2KM5Le4G5VJQFeAmcaGXb8ZTElj/vS62+LAQGG+ju?=
 =?us-ascii?Q?lhJLwetNEbmTy9I9RAFCqEdidhf7SzgZT7gf+hcuwa4oTjOJXNMRTojaJ7Ju?=
 =?us-ascii?Q?CplrnW9sqzIgjlaVi4aEUifgtRAb3Da0K/oI9NZMKg1fNH5yHiQtmCJ3xVlB?=
 =?us-ascii?Q?QNPmKcqwt0ga3npYb4VOG1AcZSYs1l38/eKBviiLBgwZc/dF9UcuKIfBbWDI?=
 =?us-ascii?Q?ZrWxJYAJMcqfP4asPBxdiMzHxZSzu2CDSRNVRjgWuZoQps7+LCbAOufPF+wE?=
 =?us-ascii?Q?H+qIxey6BcuxCbuKXGdJNGNQTLejfBuIDhuHjbrJ8msCX2/SWyJiwhGamVrd?=
 =?us-ascii?Q?spEFmvRhfCgPKY/Xvi/uH6Av6t9mTAJjcvMZRzE4?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 4f69ee32-2fb6-4b58-a7f8-08dbcae93cf7
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Oct 2023 06:05:30.3603
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: p4G+yxsVJ1tuVqUwSu3o61X34lLVFK3SFNnI0wT8aI884IRKCGcA1eotpHibLDI6pXHHT9ydgk2LjT49wV3i5A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB5265
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 11 Oct 2023 23:05:51 -0700 (PDT)

On Thu, Oct 12, 2023 at 01:36:41PM +0800, Feng Tang wrote:
> On Thu, Oct 12, 2023 at 01:23:54AM +0800, Luck, Tony wrote:
> > > IIRC, CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B was enabled in 0Day's
> > > kernel config for quite a while, to force each funtion's start
> > > address aligned on 64 bytes. Don't know if this has been changed
> > > recently.
> > >
> > > Also I noticed the patch introduce a new per-cpu variable 'storm_desc",
> > > if the function address is 64B aligned, then per-cpu data alignment
> > > may be related.
> > 
> > If adding (removing, changing) per-cpu variables can cause 8% performance
> > changes, then maybe there needs to be some way to insulate the builds
> > from these side effects (as was done with the 64-byte function alignment).
> > I've no ideas on how to achieve that :-(

As for mitigation (how to reduce these kind of strange performance
changes), I have proposed a "selective isolation" in LPC 2021, in
page 15 of https://lpc.events/event/11/contributions/895/attachments/770/1603/Strange_kernel_performance_changes_lpc_2021.pdf

As kernel binary is compactly linked together, the alignment change
of text/data in earlier modules could affect other modules after them.
MCE module is much easier to hit these strange performance changes,
just because it sit in arch/ folder and get linked early in kernel
binary.

The idea of "selective isolation" is simple, by picking some modules
sparsely and enforce some alignment to function/data/per-cpu data,
so that they can act as fences/borders, separate kernel into multiple
capsules, and make the alignment changes only take effect inside
that specific capsule.

Any thoughts? thanks!

- Feng
 
> Philip helped to clarify that 0Day kernel build does enable the 64 bytes
> function address alignment.

[...]