Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34;
Date:   Fri, 13 Oct 2023 11:28:22 -0700
From:   Ashok Raj <ashok.raj@intel.com>
To:     Linus Torvalds <torvalds@linux-foundation.org>
CC:     Linux regressions mailing list <regressions@lists.linux.dev>,
        "Thomas Gleixner" <tglx@linutronix.de>,
        Yanjun Yang <yangyj.ee@gmail.com>,
        "Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>,
        Bagas Sanjaya <bagasdotme@gmail.com>,
        "Borislav Petkov (AMD)" <bp@alien8.de>,
        Ingo Molnar <mingo@redhat.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        "the arch/x86 maintainers" <x86@kernel.org>,
        Ashok Raj <ashok.raj@intel.com>
Subject: Re: [regression] some Dell systems hang at shutdown due to "x86/smp:
 Put CPUs into INIT on shutdown if possible" (was Fwd: Kernel 6.5 hangs on
 shutdown)
Message-ID: <ZSmMRoftUrmIAeR/@a4bf019067fa.jf.intel.com>
References: <e6d1dae8-e28a-455a-a851-661c825fbdcf@gmail.com>
 <9787bd09-91b3-4a79-9ca9-e7199c925f36@leemhuis.info>
 <CAHk-=wj99i00K5ZD_OJj3d8rLG07bnTH=0_GxpzxrSzNF-WYQQ@mail.gmail.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <CAHk-=wj99i00K5ZD_OJj3d8rLG07bnTH=0_GxpzxrSzNF-WYQQ@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?M/e5M0R97Vs94gy0yUq/ClXUTW0buvVWpOSUi2FnYByMJ2xBviyoOGGtjW2h?=
 =?us-ascii?Q?k0Ft1BDwFeL8GsZ77kMa9CJwIQgTY1crPRXn4i97mjmnYF6jdeTSv6JolmK7?=
 =?us-ascii?Q?3lLkuMSKqCT4d7qwrQuvL68mKA1GaT/mWnooJznrRVZzXcPBPUc5BTpxClir?=
 =?us-ascii?Q?mMKC29qCv5RPW16GbMkVsyQde+/29yK0+ke176atG8VJgo02Vwu2TKAYclbs?=
 =?us-ascii?Q?JS57DzxL2ezoV8TSi6ZnM2pdwO8TVzDgd1PEZS0FaGrF37R++GqFn30cWjqj?=
 =?us-ascii?Q?MQACexTncgxvK+kj31RnRP3NgR/UMTn7i2EJn8pdFgOyiLElLfeQb68XGotC?=
 =?us-ascii?Q?2ExycQBqbqjjIdl8ATJIpBZ7tocNVaVRppcjruS1GiEBMOipf5jIh7g9vMQB?=
 =?us-ascii?Q?dm1S2my9EYt/sw8LMuEQndXyYzmB2aJkdgXSdLIHsGbwIsGX1UEK7oRmBb8a?=
 =?us-ascii?Q?9APmRXyjvxjwBXP7Bk3Mna+zEGfvfapvIdLyQuIPLMCpgMfpX20RdHum5r9D?=
 =?us-ascii?Q?AnJagfbPTIrmHOEIcV7HgxFpnG5j++POlol7RWgEcK4lc3k2uA8j2b2Dngml?=
 =?us-ascii?Q?KXCT3Ng7Nk/aQ5gUfhz6VHsiaMYPwtkqolEqKcCNbiRg8vg8l+CLX3aSKwxs?=
 =?us-ascii?Q?USz0MrGYG7q7OUaoUZmp+LcGs2Yq/y1D22O6Ak45ivMrWEYJGzSvEtypmU+R?=
 =?us-ascii?Q?T1sM6387/SqlW51P4WvXwXV5CCCG+rHWj2RMq4hcpf2n8SBjQe4nXTkx0xZE?=
 =?us-ascii?Q?lO7291nrJFPie7kGVDh86TyTTnkLp20hKOVJevL+FjJfuizCzG5I1+5584r0?=
 =?us-ascii?Q?wRh+kcofDqpxa6ib9rkd1SnWBj8NjQdD2iVtcPOjOrk+ehMIYNYKjWeqrIQC?=
 =?us-ascii?Q?OexHfLrSkK/bZKfzCVsY6MH2cvj0icGVGgLDV7+GQZyNi8uon0TtyH6/LI7f?=
 =?us-ascii?Q?6V9v7Ji30uRNSCa53jV0d4GTl/+r25zq3EDZwLm/sS3fItmgXSGmRU5np4A5?=
 =?us-ascii?Q?tL1MSlmWAPRHBlS+fx1/gNaonve1LyZlLaMN/r5eZ9m+GFiWTNWymUhlv2Gr?=
 =?us-ascii?Q?6NlY+E/tamtA5TYyX1tpnlflUAg8k4wva65edytRfS5ZJtdIPFmtKxdFNhRj?=
 =?us-ascii?Q?GfQWxQEEqs5NRH6UXaXtRFegLZdm35zm7BbTNEeITDXU4FyIXwCMzws7+a/p?=
 =?us-ascii?Q?W4e3GLYz/Nfe71jH02wHEmSBOCOwRcoT4eKoWr9+QtLxBmQzp/8zazROqb++?=
 =?us-ascii?Q?NyHopjrflXrkwuG5JzPC/i3qdfHD39zOmqRVOnZ0y5+5MrwaO7EVw5UAyPWF?=
 =?us-ascii?Q?fvjCNKQERDT/IL5ezSmNjOmyiu2rWlg4uQr2/RP96qBnAF/aarnV379ZfvN8?=
 =?us-ascii?Q?/THnIWNUoChtgSf3nZ4poVZokJGEvk8gi/FBE4PpmkKMSC6nJVaoN5BEgSSx?=
 =?us-ascii?Q?hYcIhOROpUiPLwdlsPZX3ZFJTLm+sprD+cEZ4E1JTQescEceOmrvy2OTXSM+?=
 =?us-ascii?Q?YRoxhA6M6uuEaNwl22MvZEr7oJyAU3LYHpJoqZfXYDeFcnPxLWOZ+Bf1xeLy?=
 =?us-ascii?Q?IyMxRWiH+Hc+L0btwExPqIRl99bhNXf6t2VIB46o?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f2e64cb1-4180-497d-2fc8-08dbcc1a322b
X-MS-Exchange-CrossTenant-AuthSource: SJ1PR11MB6201.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Oct 2023 18:28:28.7241
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 2aVBdF0TAXmMfVjswv0WPRAbJgLNDMWg8D4yTqDrtp/xauYzgM/uPzWck+bhfiVtFxsB03Xp0KiLyLpEkGCGEw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB4562
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 13 Oct 2023 11:28:54 -0700 (PDT)

Hi

On Fri, Oct 13, 2023 at 10:48:19AM -0700, Linus Torvalds wrote:
> On Fri, 13 Oct 2023 at 05:05, Linux regression tracking (Thorsten
> Leemhuis) <regressions@leemhuis.info> wrote:
> >
> > Thomas, turns out that bisection result was slightly wrong: a recheck
> > confirmed that the regression is actually caused by 45e34c8af58f23
> > ("x86/smp: Put CPUs into INIT on shutdown if possible") [v6.5-rc1] of
> > yours. See https://bugzilla.kernel.org/show_bug.cgi?id=217995 for details.
> 
> That commit does look pretty dangerous.
> 
> If *anything* is done through SMI after the code does that
> smp_park_other_cpus_in_init() sequence, I wouldn't be surprised in the
> least if the machine is hung.
> 
> That's made worse since it looks like the shutdown sequence isn't
> necessarily run on the boot CPU, so the boot CPU itself may be in
> INIT, and any SMI quite possibly ends up treating that CPU specially.

Sending INIT to processor marked as BSP will tank the system.

> 
> Who knows what SMI does, but the fact that the affected machines seem
> to be mainly from one particular manufacturer does tend to imply it's
> something like that.

There was a report (probably this same one), and it turns out it was a
bug in the BIOS SMI handler.

The client BIOS's were waiting for the lowest APICID to be the SMI
rendevous master. If this is MeteorLake, the BSP wasn't the one
with the lowest APIC and it triped here.

The BIOS change is also being pushed to others for assimilation :)

Server BIOS's had this correctly for a while now.
> 
> And the code does do a fair amount *after* shutting down cpu's. Not
> just things like calling x86_platform.iommu_shutdown(), but also
> things like possibly the tboot shutdown sequence (which almost
> *certainly* is some SMI thing).
> 
> I dunno. Thomas - I htink the argument for that commit was fairly
> theoretical, and reverting it seems the obvious thing, unless you have
> some idea of what might be wrong.
> 
>                Linus

-- 
Cheers,
Ashok