Date:   Thu, 2 Feb 2023 08:34:20 -0800
From:   Ashok Raj <ashok.raj@intel.com>
To:     Borislav Petkov <bp@alien8.de>
CC:     Dave Hansen <dave.hansen@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        LKML <linux-kernel@vger.kernel.org>, x86 <x86@kernel.org>,
        Ingo Molnar <mingo@kernel.org>,
        Tony Luck <tony.luck@intel.com>,
        Alison Schofield <alison.schofield@intel.com>,
        Reinette Chatre <reinette.chatre@intel.com>,
        Tom Lendacky <thomas.lendacky@amd.com>,
        Stefan Talpalaru <stefantalpalaru@yahoo.com>,
        David Woodhouse <dwmw2@infradead.org>,
        "Benjamin Herrenschmidt" <benh@kernel.crashing.org>,
        Jonathan Corbet <corbet@lwn.net>,
        "Rafael J . Wysocki" <rafael@kernel.org>,
        Peter Zilstra <peterz@infradead.org>,
        Andy Lutomirski <luto@kernel.org>,
        Andrew Cooper <Andrew.Cooper3@citrix.com>,
        Boris Ostrovsky <boris.ostrovsky@oracle.com>,
        Martin Pohlack <mpohlack@amazon.de>,
        Josh Poimboeuf <jpoimboe@redhat.com>,
        Ashok Raj <ashok.raj@intel.com>
Subject: Re: [Patch v3 Part2 4/9] x86/microcode: Do not call
 apply_microcode() on sibling threads
Message-ID: <Y9vmDOkDthPUx8vp@a4bf019067fa.jf.intel.com>
References: <20230130213955.6046-1-ashok.raj@intel.com>
 <20230130213955.6046-5-ashok.raj@intel.com>
 <bfd85699-4b7a-c606-266a-cea7ff336950@intel.com>
 <Y9rqeqMjDOYJKmLZ@zn.tnic>
 <Y9skuAuxozZLDrGB@a4bf019067fa.jf.intel.com>
 <Y9uFD1PQoq0Ktaol@zn.tnic>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <Y9uFD1PQoq0Ktaol@zn.tnic>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?d4baAlqALFJL8VSBSr11fehWzM4esapMbFAZ86aj+pCb5bK7RPTReL5BSny2?=
 =?us-ascii?Q?nZogZXA4I7aAn7RybLEV+qdJVa3Vvqhn0ZJmWa3SwbxnN8t0Zc/ohhijLXJ7?=
 =?us-ascii?Q?ArUJFWPUWPQrhxXdCHdkHdKKSWOHRJDU1jrAjMyzoGX/ZiWCZXVfOhzEsREH?=
 =?us-ascii?Q?DLyOdamsBWEWN6MkG2iLvJTy8UGFo1bDJ1mdzjvysra9GsNToA8lIlsvXY3d?=
 =?us-ascii?Q?6lC8bDDu7+NGXt3CpbKedO9RBwCWbv345HWX8ObaIHjpm5RsyfE8rznKYAi3?=
 =?us-ascii?Q?6M1yD3zTAFIbfyPeYdhCZsP4TkNHv/jfSnFIKSgyDim/HTQf9Ds0jKSZdHEd?=
 =?us-ascii?Q?TtQRZhmUtfJ/6u+dfT9wk3CoJXEmXVtDEPBRBxAVsWPVf1WfXR/LJTby1VVI?=
 =?us-ascii?Q?hhFZFtoOrXmnFrh6o9Uw9D9u6VFRzH2La1nenk9x2Ch9gJSQ3gF59pfD2eOO?=
 =?us-ascii?Q?wpH3rCzI/9d8qpD8IzRTgbOs4dZ3/ZIhJgy06B5jnuzCyBNRDvD1e6y6sVfC?=
 =?us-ascii?Q?I9o3kyTEzaI6k87SyorhH848vLi0t9BJClvFCGqQ3CzXZ41HDs6+ChVq+5d0?=
 =?us-ascii?Q?grzGuFtgsMvdSn8mCwJretx2y9Vy2wUJYMGLOLS9uWqOkZsd6HHKD43VIJUk?=
 =?us-ascii?Q?vQJbDM3cg0A/pb9bzkxrNmyRtQnCb/2qh9PiCvyLyWBw/kk6QpVpTOvnggHM?=
 =?us-ascii?Q?eAlMcLi3JHL5TeaT7XR/WS1zKkeNPZIBPtTqIBbq+FQgapdQM1MLDLuYoEBN?=
 =?us-ascii?Q?+HwmQrk/V2ZJRVqBkTXv9HYw8TQ+2/NBc4fGFqaRGofFe55w5BaUYIenA0HD?=
 =?us-ascii?Q?/PNBl+Tmx7m9xhcXd5dHrNwxiz2LwVNlOLw5uaMy9Uh/ok5RnUSu6WJlhQ/1?=
 =?us-ascii?Q?15Aeo4jm87bQNP11pI6JW9azxVWtT+0Wz/4/hbVsyzH1KrKOgBGGt3eJ7XF/?=
 =?us-ascii?Q?eEj+p2svShGFYmghezGJXdz9d0iamXmK2d7pp6WagnYliyDn3wqV3ZWrUuy8?=
 =?us-ascii?Q?J0rYgKLcQfP6XNnaB6fUwMXlM7+JVFIq10oM4ZNJ8dORCb6Uxo6LjVWIkAnj?=
 =?us-ascii?Q?b6hSl3Lhmo9MqAalJOFeGQrlI0s99dchedyksa8L8HoqULw4e/llwHr3OLuE?=
 =?us-ascii?Q?An8JohWpAZSm5gTOhdv5dy8IoKmt9peB5Y6Lv+OkwvoFrd2267Cb5vIXXVTI?=
 =?us-ascii?Q?tS0qPAWngBgES3jGiOn5SxNc8s2/QoGel+OMCVfF0NIOxrDnJBoFbMzbkIZ3?=
 =?us-ascii?Q?tK2HB0VTs5Fb5tAnPUIrT6heIjalxymzSAAMo812Ghf005wLk6wDX0mti5om?=
 =?us-ascii?Q?6bjXhWGasHAh6T0t5lhfy6nkwPkRunu4aYHykNkZNaEuaxMeguJaAlCTRPdg?=
 =?us-ascii?Q?oEj1JzNLnnYWbolfmd+ADqXPGtI7NoMBbWxC+OhDbbEA5AJB7582yAz5MbwR?=
 =?us-ascii?Q?UnQSdZGMQKOaF9Rud8TDAF6RTdTQsijezwqOwfbF/4gGZwv5VCJ1DxkGmN3j?=
 =?us-ascii?Q?SrPVK6F4CyhfK5oyy/mfy8l7S+nikesMsjco/1ZFC+Q6TQj/iAxyh9G58KaF?=
 =?us-ascii?Q?/g+Juw6YbFi4Ab0Sh+a3sBRC3JgxQ6zJW+7YuYKpKQBAj3o4QTKYl0uuazw6?=
 =?us-ascii?Q?OQ=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 14607505-ce42-4369-bcb7-08db053b6a10
X-MS-Exchange-CrossTenant-AuthSource: SJ1PR11MB6201.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Feb 2023 16:34:55.1102
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: GktC3imZQrebvRZv7S0k/Hz3xjSzI6xUdsi8WjUxxk4KMeF64NRKk4gcBv0kcpliJRogG2JgJ2fUoQdPfcHYsQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR11MB7047
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Thanks Boris,

On Thu, Feb 02, 2023 at 10:40:31AM +0100, Borislav Petkov wrote:
> On Wed, Feb 01, 2023 at 06:51:44PM -0800, Ashok Raj wrote:
> > T1 shouldn't be sent down the apply_microcode() path, but instead
> > just gather and update the per-cpu info reflecting the current revision.
> 
> As I said previously, it doesn't apply the microcode but simply updates
> the cached info. The assumption was that T0 would not fail.

That is super clear. But the conclusion was, in the case T0 fails there are
some side effect that are hard to manage with the current algorighm.  

That was the motivation for the  change. Sending T1 to just collect and
report the revision, but not via apply_microcode() but via some alternate
function that doesn't have any bad interaction in the possible case if T0
failed.

> 
> > At wait_for_siblings: Each thread can check their rev against the BSP (yes
> > BSP can be removed, but we can elect a core leader) and if they are
> 
> A core leader?
> 
> The one who succeeded in doing the update?

My bad, core leader is the wrong term,  I meant something like how MCE
selects a rendezvous leader today. IRC, what we call as the monarch in
MCE land when it handles broadcast MCE's.

> 
> > different we can either warn/taint or pull the plug and panic.
> 
> What about SMT=off? How are you gonna handle that? Who's the core leader
> there?

This is broken today, IRC, we started to check cpu_present_mask and
cpu_online_mask counts are equal. This change to allow was a late change.

07d981ad4cf1 ("x86/microcode: Allow late microcode loading with SMT disabled")

Since those threads are in mwait(), I'm thinking for correctness we need to
revert that change.

Reverting the patch and borking late-load if ht=off was used, seems the
mitigation to prevent that.

Most BIOS's should have a similar ht_off, and that is the best way to turn
off HT. Its probably better that way to get all execution core resources
reserved for that one active thread.

> 
> What about the other vendor? That hackery of yours needs to be verified
> there too.

You mean updating the revision only outside of the path to
apply_microcode()? 

Its only AMD/Intel today for microcode updates. Yes we should test and
validate if this assumption is correct. But that's the *assumption* today
correct?

> 
> So this whole deal needs to be analyzed properly. What would happen in
> any potential outcome, how should the kernel behave in each case, etc,
> etc.

Complete aggreement. The one case we overlooked in the current algorithm
was "what if any core is in disaggreement with each other"

If there are other cases that come to mind, I can invest in looking into
that.

> 
> In thinking about the minrev, I can just as well imagine that such
> microcode patches which could cause such a failure should be marked and
> not loaded late either. But I'm sure you don't want that.

Once there is minrev enforcement,  default action, this is automatic,
since we refuse to load anything with minrev=0. 

They automatically become eligible for early load only. 

> 
> In any case, without a proper analysis and writeup how that new
> algorithm is going to look like and behave, this is not going anywhere.
> No ad-hoc patches, no let's fix that aspect first.

I'm happy to add more into Documentation/x86/microcode.rst. I can work with
Dave Hansen on closing or enumerating the algorithm and why we made the
choices. 

Cheers,
Ashok