Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36;
Date:   Thu, 12 Oct 2023 13:36:41 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     "Luck, Tony" <tony.luck@intel.com>
CC:     "Sang, Oliver" <oliver.sang@intel.com>,
        "oe-lkp@lists.linux.dev" <oe-lkp@lists.linux.dev>,
        lkp <lkp@intel.com>,
        "linux-edac@vger.kernel.org" <linux-edac@vger.kernel.org>,
        "Huang, Ying" <ying.huang@intel.com>,
        "Yin, Fengwei" <fengwei.yin@intel.com>,
        Borislav Petkov <bp@alien8.de>,
        "Yazen Ghannam" <yazen.ghannam@amd.com>,
        "Smita.KoralahalliChannabasappa@amd.com" 
        <Smita.KoralahalliChannabasappa@amd.com>,
        "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
        "x86@kernel.org" <x86@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "patches@lists.linux.dev" <patches@lists.linux.dev>
Subject: Re: [PATCH v9 2/3] x86/mce: Add per-bank CMCI storm mitigation
Message-ID: <ZSeF6T0mkrH5pOgD@feng-clx>
References: <20231004183623.17067-3-tony.luck@intel.com>
 <202310111637.dee70328-oliver.sang@intel.com>
 <SJ1PR11MB6083D191286779302821A7EAFCCCA@SJ1PR11MB6083.namprd11.prod.outlook.com>
 <ZSbCYt35j20ezT98@feng-clx>
 <SJ1PR11MB6083154D148B42B8B7BC48D9FCCCA@SJ1PR11MB6083.namprd11.prod.outlook.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <SJ1PR11MB6083154D148B42B8B7BC48D9FCCCA@SJ1PR11MB6083.namprd11.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?7PIj0+yum5fxTfwqS/tN7M5L5FSKliTFV42dPeAEafwsRP7rcWzzxltBNTdp?=
 =?us-ascii?Q?j3+P0d+j0qsBnVmtHYC3tmLVV/lS+wKcIw8MURZweFU/COReMzbJIeAjlAPz?=
 =?us-ascii?Q?soBNvHwyoBshyFcl1LV+ptc5gaSmtXrvokOold1YVzVDre8uMP02y3GcAEiL?=
 =?us-ascii?Q?oMZ3/sBfI1X7MDUMlM7iBIxWepmgTMGEP48+lWSHSv5xOEEBEciDo3h4eXkp?=
 =?us-ascii?Q?O/uiAGB8+Vo+zxBg+i2FTGjkEvvLnBM2e57Go/YUeOJVPK/sxwvm8UkCoHmA?=
 =?us-ascii?Q?vSowfmCbpmmf0HaTLp6JFltvFgLC45kMwT/SdZBAkzqbnH/nar/jwOtEnmUD?=
 =?us-ascii?Q?+w4YvYNsPNdpGnf+WuLnQvF+bHpMErI0+KIr8qPnJyTgZ74dG63e2kEaOsuH?=
 =?us-ascii?Q?bs7nwycZiSRZJCKvk7VkMkOpcm77YtidlWDovVtA0qiW+KT5QYIv9+2VOECr?=
 =?us-ascii?Q?JXN1K+yufL1ZU3M2XC4W+PvdyrS/zgJUlVK/r4hCM/0wI4r37CrwcFcZCxrB?=
 =?us-ascii?Q?3uOUETXUjrrGPRj9heah0u2XbkXr6f+gN5vBKvHzBKak/mD2xjIrLYJOZfTq?=
 =?us-ascii?Q?NaCqSkrpIVy/wmvAyYi1wLZwKLTHjlJAbqvcHJvfddnbK9cqh+XyQQPNNDa1?=
 =?us-ascii?Q?sgXDXnLTPHyA5sXG0hFnq/922Pi2i2NQaXvc9Lven15HNcdgIM5alR6zRc5k?=
 =?us-ascii?Q?ncIDz0BxnfudlgVFLbUYUjPi45E9STR3lSbaK2cpHbuFHuKing7XFnmFeEaa?=
 =?us-ascii?Q?CfuCP361F5fP7OTBMzZBNTJ0Kxd1OrHOl1JxH6osSVjEQKOsPl0+nR8jBys4?=
 =?us-ascii?Q?FZgzKhRYHKEZ61YPOTVbg0nPnllL5sbtdbzhGbB+3tvhq9FeGn/MEbRFVnvb?=
 =?us-ascii?Q?qucIUtlkjlwbJoqAiVM9hsSO8nRujVjazVwO97zZm18zUq8rO2ZZKyAz+LvF?=
 =?us-ascii?Q?iCO5L07Q28CEZEv7/lTRe5wlamLEcA3+fO2rHitoRLVppOFzbWMoMviavEgg?=
 =?us-ascii?Q?M4BUUcXNX2IH4avugamX0aPHTlSmYikYCfWoZ4Y4yC64Zaq0mn8Ra+cP2F1n?=
 =?us-ascii?Q?PKECg+XENX7pA3sWRVEKZBwUku3G4/14BpKNJFKAEUX2H6+PebqXWT0VYYYA?=
 =?us-ascii?Q?auh64yEse310yp+sFEEw+9tEK6a0P9DEfvi30LLjhTDFzypGznUT0p5c2ZX9?=
 =?us-ascii?Q?UHXIN4acUW8AAvNTq3lofOd0etxcirOHLBjKe9d5G854U0AoPRhBwsMyQaO7?=
 =?us-ascii?Q?cR1Pup41aHiaQ3vx3ooXMxQuam1f1EPdDqtfgotfNUaiQ+P59f6vge16qdZd?=
 =?us-ascii?Q?VuYw688jmDiFC7ZvGECN4BQBlXTFQpEPrd6ZHrEUffyDC/uGlN0YjT6IyUm0?=
 =?us-ascii?Q?SS5SYNg+gF06Ny/0kepe9hggf6IjNZvyOBKuAGiEaKAFfSNAgyRgfRHcjlVO?=
 =?us-ascii?Q?yhF+z5nUY2ajASwyylpTxVnNKZUxBJ94JNGnAn8no0Ng+gXz1jxE1fwbWELP?=
 =?us-ascii?Q?6TGYn1Yu+GzNoT1oSbKEhIIAhaiFNdR/TvinlAF171V2A+5vzlWiynWq+1Is?=
 =?us-ascii?Q?CU59VJittLRfWCkcQWgZreC94rdl12OP4iYdv6pA?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ff1c5fb5-aa90-4f0e-8221-08dbcae68540
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Oct 2023 05:46:03.1683
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: M2S/THusb7bJrcYjUXytf7mKS+nXsU/G2Ax0cpGy+/mpsZy5nyX2qyErkVgO8gfWDgQPSW60gRVzJQgvspj7Ig==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7310
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 11 Oct 2023 22:47:19 -0700 (PDT)

On Thu, Oct 12, 2023 at 01:23:54AM +0800, Luck, Tony wrote:
> > IIRC, CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B was enabled in 0Day's
> > kernel config for quite a while, to force each funtion's start
> > address aligned on 64 bytes. Don't know if this has been changed
> > recently.
> >
> > Also I noticed the patch introduce a new per-cpu variable 'storm_desc",
> > if the function address is 64B aligned, then per-cpu data alignment
> > may be related.
> 
> If adding (removing, changing) per-cpu variables can cause 8% performance
> changes, then maybe there needs to be some way to insulate the builds
> from these side effects (as was done with the 64-byte function alignment).
> I've no ideas on how to achieve that :-(

Philip helped to clarify that 0Day kernel build does enable the 64 bytes
function address alignment.

> "storm_desc" is a largish per-cpu variable at 1568 bytes. But that doesn't
> quite get it into the top 20.
> 
> In the config used for my kernel build, it does appear just before a bunch
> of interrupt related per-cpu variables. Those might be sensitive to alignment?
> 
> 000000000001ff40 d smca_misc_banks_map
> 000000000001ff48 d bank_map
> 000000000001ff50 d threshold_banks
> 000000000001ff60 d storm_desc
> 0000000000020580 d tsc_adjust
> 00000000000205c0 d lapic_events
> 00000000000206c0 d vector_cleanup
> 0000000000020720 D __uv_cpu_info
> 0000000000020738 d ipi_mask

I did further check by putting 'storm_desc' into another section with
below patch, and the regression is restored, which indictes it is
related with these per-cpu data alignment change.

diff --git a/arch/x86/kernel/cpu/mce/threshold.c b/arch/x86/kernel/cpu/mce/threshold.c
index ecdf13f1bb7d5..7f32126f216bc 100644
--- a/arch/x86/kernel/cpu/mce/threshold.c
+++ b/arch/x86/kernel/cpu/mce/threshold.c
@@ -41,7 +41,7 @@ struct mca_storm_desc {
 	bool			poll_mode;
 };
 
-static DEFINE_PER_CPU(struct mca_storm_desc, storm_desc);
+static DEFINE_PER_CPU_SHARED_ALIGNED(struct mca_storm_desc, storm_desc);

Thanks,
Feng

> 
> -Tony