Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 10 Aug 2023 16:02:43 -0700
From:   Ashok Raj <ashok.raj@intel.com>
To:     Peter Zijlstra <peterz@infradead.org>
CC:     Thomas Gleixner <tglx@linutronix.de>,
        LKML <linux-kernel@vger.kernel.org>, <x86@kernel.org>,
        Borislav Petkov <bp@alien8.de>,
        Arjan van de Ven <arjan@linux.intel.com>,
        Ashok Raj <ashok.raj@intel.com>
Subject: Re: [patch 28/30] x86/microcode: Handle "offline" CPUs correctly
Message-ID: <ZNVsk2a19PuNoeSo@araj-mobl.amr.corp.intel.com>
References: <20230810153317.850017756@linutronix.de>
 <20230810160806.562016788@linutronix.de>
 <20230810204605.GF212435@hirez.programming.kicks-ass.net>
 <ZNVNibrpZ9bJLok7@araj-mobl.amr.corp.intel.com>
 <20230810210511.GH212435@hirez.programming.kicks-ass.net>
 <ZNVbc7qy8k49Dwhi@araj-mobl.amr.corp.intel.com>
 <20230810222957.GJ212435@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230810222957.GJ212435@hirez.programming.kicks-ass.net>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?D58KupdptL4PvvmpKQPAVQYZydCk5Np2AiHrwnaqf2PH2NZEeyZnmnSWOM8E?=
 =?us-ascii?Q?zs2LiyXKWw40HrG1j730knXZOyiyIid15V/HodpwsaoPhfBhBPfFF6PDKfPD?=
 =?us-ascii?Q?1xut8inqGord1b39OY2AvGfQFG/8MBzXOKsT3rqM/AI/0tqI3OfjMEcOHRh1?=
 =?us-ascii?Q?yOBS3y/q8yG4kISA/rfFSAU/Ysy/ULI6Gl50e5HIGlrBhjfFtOnXE00okGvs?=
 =?us-ascii?Q?iptIe4y7U849+mEhgEK4IecrMc8RfzctjbnJj6oepqjZkKivaOofN+/XoAbH?=
 =?us-ascii?Q?le8iVMppksP3Anut27jstKhyG1v+zfjfN86OM2Dbdd3kZgaHR2MauvezYHjC?=
 =?us-ascii?Q?hL8R2J95aA8USP75SFJZQq17twomAtWVNuXhpaOAvXbNis8SRTr9EPyDQbF2?=
 =?us-ascii?Q?9RkZL13Ez96w4o1oUyV7T/u1V9HPXBDfEmS0DvV5Gdr+D3riWMegd4X5TR5L?=
 =?us-ascii?Q?Gw2+7S7fjOZQ9jPCiCjKRLEpMxxmf+uoYX+/AOALRNu4TA+b5r81BlKrIlJM?=
 =?us-ascii?Q?SK4F0N7FvZeaCXUVtkOdiY7Zlfh2WBWqiCFF5r5t/ZBELPe+PCwWKrAwsEEY?=
 =?us-ascii?Q?24MWt4NJ65y8Jw4qtyzH0gjjJxblKcxGzNSJvcvLsTy4SvnkHISlZE1nt/4/?=
 =?us-ascii?Q?gk4i9MGeqk4cNdij03Tae7MPRRLk6pAA2/jUq5Jucn03rPPyzMAfyrbsEj/F?=
 =?us-ascii?Q?dFghs+GlqJqNaN21Z81wpat+SDu+iuMkhip4BH9PLqrBWj0pAtyQoNacV0s0?=
 =?us-ascii?Q?yKobnflsQfkhw5HfebygdPYU3A/jlY8L4OeYfTmttCPR5zkgZgh/e2Op1rL2?=
 =?us-ascii?Q?0h85/V9gckVB2LXnnnpM9bd4hhPl0lxkrHL0Alku8QFK7F4gji65zEGst+Lm?=
 =?us-ascii?Q?yGpj8GOgjhkSO6aIkBmTAnBnlYFsIRvr6i5mSMUUhzBomQkVsbD1pU41igOf?=
 =?us-ascii?Q?zvgy0JR4VL4WLyKIQk/p5S1HH+N+nwC47hslCX1UEQCcvH73nJN3aeSYwnqH?=
 =?us-ascii?Q?hYmCZVdEBcCEOjViqRc8Es/hIyjZEUzG777ANetYluSkfZGVz6ZymShL27In?=
 =?us-ascii?Q?M10sckTgOV2lio4Hy3VRD+ToSL/Uj63UpPgtONSURjoFad9/UcIz+BXONerC?=
 =?us-ascii?Q?pTMfxUFBsp4lTI5tfs28c7FdY/ERavDWMMNCGARp1swr8VI09fH9iBsWzvJr?=
 =?us-ascii?Q?T2DuQHaWjKyywlkF0XE2sGAulqK1to01L5cIPs7RAnK1XzNYtklRrpycfoJI?=
 =?us-ascii?Q?q0BjOBM1EAGiARvIMRhym3R3UtIbm/FVE7Eq6tYbr/f9hHcoBugVr9MtmGN/?=
 =?us-ascii?Q?T4D9y3eXZIDupchMVK5WzsPU/rlo5oo6A+Bsj1LJt7KcplPBcNDjMoyYeutd?=
 =?us-ascii?Q?342pthYdanclBjN8aJmSNFWMB0HHcg4m8rFwvLLGDTvygj4/IfjrYwSKd/Ds?=
 =?us-ascii?Q?YDjtXJKKN132cBgBngFos2YloNRfFeZKDs0Nzk3E5RC0fiD1JEncUNj9xvG7?=
 =?us-ascii?Q?rNy92PFuuO8+3svCnLnpesUZOXwfAzd95RDuZPhtRvRS8BvVh46lg1GECBWB?=
 =?us-ascii?Q?cxvvZ+UcfAUkqoyMok+0rCES+A8PfYogtay3O71u?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1574ac58-0fd8-44cc-b386-08db99f5e9e7
X-MS-Exchange-CrossTenant-AuthSource: SJ1PR11MB6201.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Aug 2023 23:02:47.4784
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: QC60C6/SfEbtTISzglWcmvt8fYraGNG11KXhta0wQus32ra/BZhigPMFT8PG17BHRBXIZ/XNp8nHvW5sf3i08A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR11MB8680
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Aug 11, 2023 at 12:29:57AM +0200, Peter Zijlstra wrote:
> On Thu, Aug 10, 2023 at 02:49:39PM -0700, Ashok Raj wrote:
> > On Thu, Aug 10, 2023 at 11:05:11PM +0200, Peter Zijlstra wrote:
> > > On Thu, Aug 10, 2023 at 01:50:17PM -0700, Ashok Raj wrote:
> > > > On Thu, Aug 10, 2023 at 10:46:05PM +0200, Peter Zijlstra wrote:
> > > > > On Thu, Aug 10, 2023 at 08:38:07PM +0200, Thomas Gleixner wrote:
> > > > > 
> > > > > >  	for_each_cpu_and(cpu, cpu_present_mask, &cpus_booted_once_mask) {
> > > > > > +		/*
> > > > > > +		 * Offline CPUs sit in one of the play_dead() functions
> > > > > > +		 * with interrupts disabled, but they still react on NMIs
> > > > > > +		 * and execute arbitrary code. Also MWAIT being updated
> > > > > > +		 * while the offline CPU sits there is not necessarily safe
> > > > > > +		 * on all CPU variants.
> > > > > > +		 *
> > > > > > +		 * Mark them in the offline_cpus mask which will be handled
> > > > > > +		 * by CPU0 later in the update process.
> > > > > > +		 *
> > > > > > +		 * Ensure that the primary thread is online so that it is
> > > > > > +		 * guaranteed that all cores are updated.
> > > > > > +		 */
> > > > > >  		if (!cpu_online(cpu)) {
> > > > > > +			if (topology_is_primary_thread(cpu) || !allow_smt_offline) {
> > > > > > +				pr_err("CPU %u not online, loading aborted\n", cpu);
> > > > > 
> > > > > We could make the NMI handler do the ucode load, no? Also, you just need
> > > > > any thread online, don't particularly care about primary thread or not
> > > > > afaict.
> > > > 
> > > > Patch 25 does that load in NMI. You are right, we just need "a" CPU in each
> > > > core online. 
> > > 
> > > Patch 25 does it for online CPUs, offline CPUs are having a separate
> > > code path:
> > > 
> > >   microcode_nmi_handler()
> > > 
> > > vs
> > > 
> > >   microcode_offline_nmi_handler()
> > 
> > Since the code enforces all primary CPUs to be ONLINE, the secondaries are the
> > other thread of the same core. So they automatically get the update when
> > primary does it. 
> > 
> > The secondaries are parked in NMI just to avoid the risk of executing code
> > that might be patched by primary.
> > 
> > Or maybe you had something else in mind. 
> 
> Yeah, not placing constraints on who is online at all. Also, if both
> siblings are offline, then onlining will re-load ucode anyway, no?

We need one thread in a core online,  because a MCE can happen and we don't
want those running something stale.