Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 25 Apr 2023 17:10:42 -0700
From:   "H. Peter Anvin" <hpa@zytor.com>
To:     Dave Hansen <dave.hansen@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Tony Battersby <tonyb@cybernetics.com>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>, x86@kernel.org
CC:     Mario Limonciello <mario.limonciello@amd.com>,
        Tom Lendacky <thomas.lendacky@amd.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Andi Kleen <ak@linux.intel.com>
Subject: Re: [PATCH RFC] x86/cpu: fix intermittent lockup on poweroff
User-Agent: K-9 Mail for Android
In-Reply-To: <ecdea7a8-a748-6ecb-5fc1-93d7eda3c54d@intel.com>
References: <3817d810-e0f1-8ef8-0bbd-663b919ca49b@cybernetics.com> <f5c7a104-d422-bd02-d361-e9e9f433d41d@intel.com> <87o7nbzn8w.ffs@tglx> <ecdea7a8-a748-6ecb-5fc1-93d7eda3c54d@intel.com>
Message-ID: <D9668AEC-9650-4840-A03D-553994B414DD@zytor.com>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On April 25, 2023 3:29:49 PM PDT, Dave Hansen <dave=2Ehansen@intel=2Ecom> w=
rote:
>On 4/25/23 14:05, Thomas Gleixner wrote:
>> The only consequence of looking at bit 0 of some random other leaf is
>> that all CPUs which run stop_this_cpu() issue WBINVD in parallel, which
>> is slow but should not be a fatal issue=2E
>>=20
>> Tony observed this is a 50% chance to hang, which means this is a timin=
g
>> issue=2E
>
>I _think_ the system in question is a dual-socket Westmere=2E  I don't se=
e
>any obvious errata that we could pin this on:
>
>> https://www=2Eintel=2Ecom/content/dam/www/public/us/en/documents/specif=
ication-updates/xeon-5600-specification-update=2Epdf
>
>Andi Kleen had an interesting theory=2E  WBINVD is a pretty expensive
>operation=2E  It's possible that it has some degenerative behavior when
>it's called on a *bunch* of CPUs all at once (which this path can do)=2E
>If the instruction takes too long, it could trigger one of the CPU's
>internal lockup detectors and trigger a machine check=2E  At that point,
>all hell breaks loose=2E
>
>I don't know the cache coherency protocol well enough to say for sure,
>but I wonder if there's a storm of cache coherency traffic as all those
>lines get written back=2E  One of the CPUs gets starved from making enoug=
h
>forward progress and trips a CPU-internal watchdog=2E
>
>Andi also says that it _should_ log something in the machine check banks
>when this happens so there should be at least some kind of breadcrumb=2E
>
>Either way, I'm hoping this hand waving satiates tglx's morbid curiosity
>about hardware that came out from before I even worked at Intel=2E ;)

"Pretty expensive" doesn't really cover it=2E It is by far the longest tim=
e an x86 CPU can block out all outside events=2E