Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Sun, 14 Aug 2022 14:41:18 +0000
From:   Ashok Raj <ashok.raj@intel.com>
To:     Andrew Cooper <Andrew.Cooper3@citrix.com>
CC:     Andy Lutomirski <luto@kernel.org>, Borislav Petkov <bp@alien8.de>,
        "Thomas Gleixner" <tglx@linutronix.de>,
        Tony Luck <tony.luck@intel.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        the arch/x86 maintainers <x86@kernel.org>,
        "luto@amacapital.net" <luto@amacapital.net>,
        Tom Lendacky <thomas.lendacky@amd.com>,
        Ashok Raj <ashok.raj@intel.com>
Subject: Re: [PATCH 5/5] x86/microcode: Handle NMI's during microcode update.
Message-ID: <YvkJjsdlDIcerqLg@araj-dh-work>
References: <20220813223825.3164861-1-ashok.raj@intel.com>
 <20220813223825.3164861-6-ashok.raj@intel.com>
 <0d6afdb6-e546-4ef0-80c2-84827c11f2ce@www.fastmail.com>
 <Yvhj5YaLdJnF60uR@araj-dh-work>
 <328bf268-51b3-f9b2-1315-bac0e883d135@citrix.com>
Content-Type: text/plain; charset="iso-8859-1"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <328bf268-51b3-f9b2-1315-bac0e883d135@citrix.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?EiTygCLo2aOchxYJ6Y5IY7K+r/8T2VUShh3Nymd3gioXfhxicsC2HxiP4K?=
 =?iso-8859-1?Q?BqtEi2FQ/MnHvfwkP9fJrasvzI148XhF0zhRkmCCz95Gvzb9sSJP5qVlHr?=
 =?iso-8859-1?Q?tPmVStaNvNEZSEMyDXqi1TWDf6E7Xrvev9RpaFAfwnjKROER+ZzpbfXv60?=
 =?iso-8859-1?Q?jVgrizLKWUqHBljHMSPVo9unbzjs1eTUbyTRRGfi8kg+5MGaKn6+edP6FN?=
 =?iso-8859-1?Q?CRuSVYKXORpxrP1trIQV+Wp8g+X50uDpUGZBQ0CXlvj3muXSeaUdwKXkXn?=
 =?iso-8859-1?Q?xzE4jICswrLH9Ls8Bwj0gVq8s5loIPQHWstu4QQS+W1SgwmsI6YIhPj/aE?=
 =?iso-8859-1?Q?FoTUAvwm59O8RxfztY/8V3KIQLfnOleyXl4TSeh2SmNJOCOwkZuSS5nUz2?=
 =?iso-8859-1?Q?TpuVVw8f5PFPVkJ78LMUAaXnnvnv4qsTovTAwiy4zU+RSjbxvBanm4DXIB?=
 =?iso-8859-1?Q?ptt/lhK83+h7WEGbEkXjlMi6CoFCxhpU/A4V4FbLxBooMuk2aZmG94gkID?=
 =?iso-8859-1?Q?g+I6yGOELsNhOEXBWsxPZGff6SMH3s6/DTJnXgmEwAUj1KN0xgdTLQ7uf9?=
 =?iso-8859-1?Q?VO2Itz5eJHm5lE21gjIzuWeb3lGXaU00Op/c1R/OAQ7yx5iYbKLgHoqbu0?=
 =?iso-8859-1?Q?TAX/ivlLvpVGz4Tfxm+HyfuwDMTCFAtiIEEbv1QdYvAwH/CYGvv8VFCpH6?=
 =?iso-8859-1?Q?pNx0YWv1FXrkWiaurpjIh1RGmFU9LZse3g9T5MVvHyvsviXISOppQSlEK5?=
 =?iso-8859-1?Q?LlBCKnfm1TVjP3OJXFo+iSgxE74DDCrXpGY1sEgZ8sO3wKxOUHS9KmAh/d?=
 =?iso-8859-1?Q?lDhQsuJVhe3XGaW0Yf2+Rx3xTOddKKnqcb/AYbEHaZWaFA1hr9Io6AKf9L?=
 =?iso-8859-1?Q?Bmj1Yj8Qhf9t15t7RJNpIkOVfESoqPua6KCjuhxxFx3qE0FWuBjAEqoaJp?=
 =?iso-8859-1?Q?Lv+bNjUXYDGUqtDlMgq/JmP5YNftXTE0ibVeo2P2KsQmzLxpK2XEVrq6Xn?=
 =?iso-8859-1?Q?D1Ub4k/0q3NZuKYx6+pkWG33KMEvphxoGV8qbF3hdcV6/EGhKqgfK8cIan?=
 =?iso-8859-1?Q?xqgHtoyZheg+LENqX6xUEXmm+60X/4j3wEgByiqn4esN8L8Dv4cmvih5hw?=
 =?iso-8859-1?Q?KS8Sj2rpJ3gvvScGRvwlkMYAmtOQGgBRXZDW5jhBOlEWupdzfRJfzJiZdH?=
 =?iso-8859-1?Q?0bo5EPBkG3h5EpLpH9hz7+LqG/19gF7p4TB4LDI0Tke5vRXIcnJeaTtFtX?=
 =?iso-8859-1?Q?A0rdMQAg2pv7uYaNFIGrGFEKzRm8OH39laHc0qhGjukseWqwFx+H7nogzr?=
 =?iso-8859-1?Q?Vi/PNqKCY/8MxDzLe48WpDb1dKXHran6QQXIj4pnZjZD4caGtKia2yG8nM?=
 =?iso-8859-1?Q?+TQdr6IvaVpazPclAsOLQEsggOwTdteLfoyCa0xNWeKcAybGmIIZ0Ji8QI?=
 =?iso-8859-1?Q?3ycdwwoXn6Tz6XHDJkCK7L4hhMcbe8+qJGVyhGvrJtel+Qt9z9Pnai1WQZ?=
 =?iso-8859-1?Q?OBDsw50RUi7VHUCy3qGavapztP3sxR5qiv+awhmdwdm5edIbVMQOvB2LSo?=
 =?iso-8859-1?Q?JpgQr3TMWE1M/G2y90LsuYICh8faU9Pv+AZHxzZaO9DvM8en9WnfUldZkJ?=
 =?iso-8859-1?Q?XZoiOzHz19izypKHq6+W0frP0R9fkeh9okPhkyHEN8syClGkxILD79Lw?=
 =?iso-8859-1?Q?=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 0841676f-f7cd-4ca6-40e9-08da7e031716
X-MS-Exchange-CrossTenant-AuthSource: SJ1PR11MB6201.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2022 14:41:36.5569
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 4mRjn0DGANPyZ/yL/x77u8jC/kCLX5F5ln8F+ojq/hUghl7iuBoBBq5fbrZ+TZj/hgP9wYRWVtuT+gSoCUSLMg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR11MB4301
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Andrew,

On Sun, Aug 14, 2022 at 11:58:17AM +0000, Andrew Cooper wrote:
> >> If I were implementing this, I would rendezvous via stop_machine as usual.  Then I would set a flag or install a handler indicating that we are doing a microcode update, send NMI-to-self, and rendezvous in the NMI handler and do the update.
> > Well, that is exactly what I did for the first attempt. The code looked so
> > beautiful in the eyes of the creator :-) but somehow I couldn't get it to
> > not lock up.
> 
> So the way we do this in Xen is to rendezvous in stop machine, then have
> only the siblings self-NMI.? The primary threads don't need to be in NMI
> context, because the WRMSR to trigger the update *is* atomic with NMIs.
> 
> However, you do need to make sure that the NMI wait loop knows not to
> wait for primary threads, otherwise you can deadlock when taking an NMI
> on a primary thread between setting up the NMI handler and actually
> issuing the update.
> 

I'm almost sure that was the deadlock I ran into. You are correct, the
primary thread doesn't need to be in NMI, since once the wrmsr starts,  it
can't be interrupted.

But the primary needs to wait until its own siblings have dropped into NMI.
Before proceeding to perform wrmsr.

in stop_machine() handler, primary thread waits for its thread siblings to
enter NMI and report itself. Siblings will simply self IPI and then proceed
to wait for exit_sync

then primary does the wrmsr flow
clears the wait_cpus mask so that secondary inside NMI hander can release
itself

resync at exit rendezvous.

I have this coded, will test and repost.

Cheers,
Ashok