Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
From:   Thomas Gleixner <tglx@linutronix.de>
To:     Linus Torvalds <torvalds@linux-foundation.org>,
        Linux regressions mailing list <regressions@lists.linux.dev>
Cc:     Yanjun Yang <yangyj.ee@gmail.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Bagas Sanjaya <bagasdotme@gmail.com>,
        "Borislav Petkov (AMD)" <bp@alien8.de>,
        Ashok Raj <ashok.raj@intel.com>,
        Ingo Molnar <mingo@redhat.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        the arch/x86 maintainers <x86@kernel.org>
Subject: Re: [regression] some Dell systems hang at shutdown due to
 "x86/smp: Put CPUs into INIT on shutdown if possible" (was Fwd: Kernel 6.5
 hangs on shutdown)
In-Reply-To: <CAHk-=wj99i00K5ZD_OJj3d8rLG07bnTH=0_GxpzxrSzNF-WYQQ@mail.gmail.com>
References: <e6d1dae8-e28a-455a-a851-661c825fbdcf@gmail.com>
 <9787bd09-91b3-4a79-9ca9-e7199c925f36@leemhuis.info>
 <CAHk-=wj99i00K5ZD_OJj3d8rLG07bnTH=0_GxpzxrSzNF-WYQQ@mail.gmail.com>
Date:   Fri, 13 Oct 2023 21:40:08 +0200
Message-ID: <87pm1i71lz.ffs@tglx>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: bulk

On Fri, Oct 13 2023 at 10:48, Linus Torvalds wrote:
> On Fri, 13 Oct 2023 at 05:05, Linux regression tracking (Thorsten
> Leemhuis) <regressions@leemhuis.info> wrote:
>>
>> Thomas, turns out that bisection result was slightly wrong: a recheck
>> confirmed that the regression is actually caused by 45e34c8af58f23
>> ("x86/smp: Put CPUs into INIT on shutdown if possible") [v6.5-rc1] of
>> yours. See https://bugzilla.kernel.org/show_bug.cgi?id=217995 for details.
>
> That commit does look pretty dangerous.
>
> If *anything* is done through SMI after the code does that
> smp_park_other_cpus_in_init() sequence, I wouldn't be surprised in the
> least if the machine is hung.
>
> That's made worse since it looks like the shutdown sequence isn't
> necessarily run on the boot CPU, so the boot CPU itself may be in
> INIT, and any SMI quite possibly ends up treating that CPU specially.

smp_park_other_cpus_in_init() bails out early when it's not invoked on
the boot CPU because sending INIT to the BSP results in a full machine
reset. So that's definitely not the problem.

> Who knows what SMI does, but the fact that the affected machines seem
> to be mainly from one particular manufacturer does tend to imply it's
> something like that.

It's mostly DELL machines. The rest seems to be Lenovo and Sony with
Alderlake/Raptorlake CPUs - at least that's what I could figure out from
the various bug reports. I don't know which CPUs the DELL machines have,
so I can't say it's a pattern.

Bagas, can you please provide the output of /proc/cpuinfo ?

> And the code does do a fair amount *after* shutting down cpu's. Not
> just things like calling x86_platform.iommu_shutdown(), but also
> things like possibly the tboot shutdown sequence (which almost
> *certainly* is some SMI thing).

That should not matter, but who the heck knows.

> I dunno. Thomas - I htink the argument for that commit was fairly
> theoretical, and reverting it seems the obvious thing, unless you have
> some idea of what might be wrong.

I agree with the revert for now.

The problem is not entirely theoretical in the kexec() case, but yes for
shutdown/reboot it's irrelevant.

The reason why I ended up with this is the initial problem of soft
offlined CPUs sitting in MWAIT. The kexec() kernel can end up writing to
the monitor cache line reliably after it overwrote the original kernel
mappings, which results in completely undebugable chaos or triple
faults.

The MWAIT issue is mitigated by writing to the monitor cache lines and
forcing the CPUs into HLT.

Extensive testing revealed that HLT is not entirely safe either, so we
ended up with the INIT trick, which turned out to be very reliable in
testing. Though it's obviously making some BIOSes very unhappy. Sigh...

Did I mention before that I hate computers with a passion?

Thanks,

        tglx