Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <ecdea7a8-a748-6ecb-5fc1-93d7eda3c54d@intel.com>
Date:   Tue, 25 Apr 2023 15:29:49 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.10.0
Subject: Re: [PATCH RFC] x86/cpu: fix intermittent lockup on poweroff
Content-Language: en-US
To:     Thomas Gleixner <tglx@linutronix.de>,
        Tony Battersby <tonyb@cybernetics.com>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>, x86@kernel.org
Cc:     "H. Peter Anvin" <hpa@zytor.com>,
        Mario Limonciello <mario.limonciello@amd.com>,
        Tom Lendacky <thomas.lendacky@amd.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Andi Kleen <ak@linux.intel.com>
References: <3817d810-e0f1-8ef8-0bbd-663b919ca49b@cybernetics.com>
 <f5c7a104-d422-bd02-d361-e9e9f433d41d@intel.com> <87o7nbzn8w.ffs@tglx>
From:   Dave Hansen <dave.hansen@intel.com>
In-Reply-To: <87o7nbzn8w.ffs@tglx>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 4/25/23 14:05, Thomas Gleixner wrote:
> The only consequence of looking at bit 0 of some random other leaf is
> that all CPUs which run stop_this_cpu() issue WBINVD in parallel, which
> is slow but should not be a fatal issue.
> 
> Tony observed this is a 50% chance to hang, which means this is a timing
> issue.

I _think_ the system in question is a dual-socket Westmere.  I don't see
any obvious errata that we could pin this on:

> https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-5600-specification-update.pdf

Andi Kleen had an interesting theory.  WBINVD is a pretty expensive
operation.  It's possible that it has some degenerative behavior when
it's called on a *bunch* of CPUs all at once (which this path can do).
If the instruction takes too long, it could trigger one of the CPU's
internal lockup detectors and trigger a machine check.  At that point,
all hell breaks loose.

I don't know the cache coherency protocol well enough to say for sure,
but I wonder if there's a storm of cache coherency traffic as all those
lines get written back.  One of the CPUs gets starved from making enough
forward progress and trips a CPU-internal watchdog.

Andi also says that it _should_ log something in the machine check banks
when this happens so there should be at least some kind of breadcrumb.

Either way, I'm hoping this hand waving satiates tglx's morbid curiosity
about hardware that came out from before I even worked at Intel. ;)