Received-SPF: pass (google.com: domain of linux-kernel+bounces-147724-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <77fe7722-cbe9-4880-8096-e2c197c5b757@oracle.com>
Date: Tue, 16 Apr 2024 18:56:28 -0400
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] KVM/x86: Do not clear SIPI while in SMM
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: kvm@vger.kernel.org, seanjc@google.com, linux-kernel@vger.kernel.org
References: <20240416204729.2541743-1-boris.ostrovsky@oracle.com>
 <c7091688-8af5-4e70-b2d7-6d0a7134dbbe@redhat.com>
 <66cc2113-3417-42d0-bf47-d707816cbb53@oracle.com>
 <CABgObfZ-dFnWK46pyvuaO8TKEKC5pntqa1nXm-7Cwr0rpg5a3w@mail.gmail.com>
Content-Language: en-US
From: boris.ostrovsky@oracle.com
In-Reply-To: <CABgObfZ-dFnWK46pyvuaO8TKEKC5pntqa1nXm-7Cwr0rpg5a3w@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?Nmc5Q1ZjNHB3Snp2NFlHUlpVdUQwODBWank0V1QzU3o0TTZsajBoNVRSSjRz?=
 =?utf-8?B?bk5uNnk5dDBWb2JpY1VwR05HaEZRV2Eyek15d1ZjaFhmNy9YTDRYb0daUVNW?=
 =?utf-8?B?cnl5SmM3U2lhNGZ4TmdmZTY5VDVnVDlNNWoyUmFFbUVoZktlZUQ0TnUyOHZ1?=
 =?utf-8?B?RU9TZ3BxdUlUU1BFR1lRN0VDbEk4K0tLM2RhMG95VS92MXkxaHQrTkhDbHFV?=
 =?utf-8?B?LzhqRzNsVmFWa0RZSHRPc2F5UXFIQ3pzQUZhUWtJWWhiZTk2UU5yZVlyQXZj?=
 =?utf-8?B?MWdtbkNVbFNoaW9IcC9yS3RLb3VWdmJjendjYXVZWWJNL2RPOVhoY1hVMEQr?=
 =?utf-8?B?cXlHKzRoQjVyN0dMTzN1enJKMm1zUEY5TjZrK0swY0t3SlZONU5wQUthTGZ1?=
 =?utf-8?B?bEZLU09uQUllQ2RMc2xDaFE2Q1A3MWRBT3ZjWWNicXZvdmoxK3VZQ2tLa05H?=
 =?utf-8?B?NGlhYTk0NjlSUFdXSitkUjlIMjU5V0RqYVlmMi8zRmNXdHZrcGU1eHQ1NmFI?=
 =?utf-8?B?NlFYTGdwVEdWSzMrQnQ3QWFTdk5qUUVScmY3MW9PanhaWEt4S0JOWndqc0tl?=
 =?utf-8?B?WWhxZTBFckxkTUlpYU96SGVma3lkWFhIZlFZSXNKWEF0THMvN1R0NzYzZndS?=
 =?utf-8?B?dnNqS2l5c2l1Q3BwMnhSb1g4Z0ZhS2IwaTVRa0lheDhocFBhUVNqeDRZanBM?=
 =?utf-8?B?K3JUeWNzUkx6ZFNkTTNKQkFSWkVzQkc0elh5N2JpU2xQSlJSYkNIaWRUQlV4?=
 =?utf-8?B?eUpPbTlhN09UaE5UTHVFeWgyNGVEVW80My9vaXlKeVRIaTRsTTY4ZW9MQm0v?=
 =?utf-8?B?emc2VlMrV2dvSDZBUVYzWGtmN3d5bGNBalJ4elord3oxbVRUNHJsenFpQUQx?=
 =?utf-8?B?K1djaEt4cWRGbE4rQWZmVW45RzlUd1I5cXVWYmQzSFJ3WFFXQkg0VUJIaGpp?=
 =?utf-8?B?VEd6b1lCQlJRK3lhZ3lPamo1d010NTdEdFlSMUd0aUpDMlE3Mm82QUd4RTJ6?=
 =?utf-8?B?a1JOdGF6WjVZOGF3SFZISzFJWUlHZUpmaHJuUUwzbEFXd282cE5aQ3M3T2d2?=
 =?utf-8?B?MklpQWlNa0FReHRrU3pLTVlJSjBmK0RwL01KbzFQK2g3K3RRYlpRUmhoK0Qr?=
 =?utf-8?B?SGpkMHJjUVc0aUViV0d4UVNGanJOMCtPYk5mSC9FVnA5U2xTaGl6b055TUxW?=
 =?utf-8?B?VjA4Wmk0ais0NERVZnRPOGhWZU4xcytod3FkRFNmNkZDUUhoZVN5bmNkZHRs?=
 =?utf-8?B?TVllK2RRWEcvTEZhTUlmbXBkTHBjRGVLR20rcW03RlJ1V0V6TE1PVm1xbEVE?=
 =?utf-8?B?K3pVYWNhM3p3Szg2V2o3YTdtVFJKQ2ppT1hhSjFIZVhGNE1BMnlMdTVySXBJ?=
 =?utf-8?B?U29xTnU5dDNXSGoxeml4dmFGZGZrc3AvOTJ0Y3JwbmNua0svbXFJVmtqSEJY?=
 =?utf-8?B?TkJwcEVjdnYxT0xiTkRWOFRPTm5QbDJ1bGd0WGM1OUY0NXJnNTdGK3VsQ0wr?=
 =?utf-8?B?dUsyUGVQcXBaQ2ZvYXRKQ1FuZ1RMTVNER2ZwKzhTWE1JUGtiaUdnMmEzTVFz?=
 =?utf-8?B?anoyUUhINTg3RlRaUUoyZmZPaDZjaEQ3NFplL0x4dm4rY3BwaHJTdlEwK1JN?=
 =?utf-8?B?ck9OMitqMnUzaHhlRmEwbW9IQ3phSzdGVGttRmQ5dDNCNTVLTytrM1hmNXdH?=
 =?utf-8?B?NXJDSU11NUJrdit0S0pJWWpUN0pNNyt4ejhkYTdEcHBsaDdhdG15SzIvQWEy?=
 =?utf-8?B?RkdIM3RBNkVJVWwrTmFFOEdZQ2ZBRTRVRFIyUG5iN3pJTXJmSEpGUVFZV1FQ?=
 =?utf-8?B?dGhjYU82SWw5ZlVWOHV4ek5XczFtMnpPS3hkOUgwaHc5WUpyaVpCbUJKVXMr?=
 =?utf-8?B?Znc3alh4ZzhuZXZmU09EbkoxYnBpazBJMlR4VkdJYzlnUEIvWmFvV1J6VUlF?=
 =?utf-8?B?bWpkTFlsYzBPcnF3WXN4NEhqcTNxbFo3d3ZoNVUyU1lHSDFiMEthVGkwQW5r?=
 =?utf-8?B?RHBKV1ZINm5zMW5OVDZyNFdnc0s1VmgySXIydmNSeDd3UVBWa1l0bFk0MGRu?=
 =?utf-8?B?c3FsaTlHVlFmM0VmczQ2ODI3QUNJTDJhakdJWjhVcEtPV3l4KzNYUjVYMEJQ?=
 =?utf-8?B?amV0R1h3U1M1SHdGZmpTWDhCdGxnQWZocXJwR2diWHJ3NlE5NWV3bXdyT2hP?=
 =?utf-8?B?WFE9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	O7BPgcWps/XoZUh4I4k7OLtLZlMSZaC5qbUNycXLTzcUzYi0eiMIphVfzDdIZmf45fWeUmMirEvehx/BgbRqLJXSxXAxG4jFcz/ASgoYlwPTFQALw10TVQM0WlnaP77vaogPXg9I8DC5gKsUL/j6ITmsZmbkWbGTmrGrHahsG/RkgiYHmLxhvbpKJn3GvExFqR6yI1cu/3LgyMeCp5saQzVJ46aN6M7zvCAUzVGNUjzsDs04PuKMXwbp9Z2kO9bDOenvS/odlbR8QTk7+C+LutRhPU2NJrXKt7xxXvWPraTj8ub63+4uA0KHmwwVqRtfs0Xyakl8gk5IOkKOQHXXgiZQadcz03SjH6TEqBQKeb5VyFzowO9m4y8i3gPTemWeR3Z2huY9XrrhRuUu5qeGEV7zaybyCUTZYdj9LrfhhCpzjs8TqTt1ho78sSw4NmZN/IacON0iVQi5k0GDmkI/Hj+GaKPcnU8zHyE58lXUBkIjhbz0bTS/gOaOyRgdqc3NMb9S2wdNYtXwtz2Qiq69cwDGrw4lqGINx41RohsjjBAWc4e0ePr3cDcyMgAqKJPetrFDrPd3+5DNu4mi9+/BpujIuayJaqtpJc/ybBdBZh0=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6503babd-2db8-4b73-025f-08dc5e687739
X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB5009.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Apr 2024 22:56:35.0901
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: F7T+USXCmm8RY4IoAwef8iaGLJRTdgJIqOxabfQ7I5kchCSMcAcVlTn430NNHzVca8D1sXVisVDkeggYSsrEQOTU4sC+vPPAplnBHc6vi5s=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR10MB4230
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-04-16_18,2024-04-16_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0
 mlxlogscore=999 phishscore=0 mlxscore=0 malwarescore=0 bulkscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2404010000 definitions=main-2404160149
X-Proofpoint-ORIG-GUID: Zb_ZD1AMiMbMkCNsbweHJZXW-TkTmad0
X-Proofpoint-GUID: Zb_ZD1AMiMbMkCNsbweHJZXW-TkTmad0

(Sorry, need to resend)

On 4/16/24 6:03 PM, Paolo Bonzini wrote:
> On Tue, Apr 16, 2024 at 10:57 PM <boris.ostrovsky@oracle.com> wrote:
>> On 4/16/24 4:53 PM, Paolo Bonzini wrote:
>>> On 4/16/24 22:47, Boris Ostrovsky wrote:
>>>> Keeping the SIPI pending avoids this scenario.
>>>
>>> This is incorrect - it's yet another ugly legacy facet of x86, but we
>>> have to live with it.  SIPI is discarded because the code is supposed
>>> to retry it if needed ("INIT-SIPI-SIPI").
>>
>> I couldn't find in the SDM/APM a definitive statement about whether SIPI
>> is supposed to be dropped.
> 
> I think the manual is pretty consistent that SIPIs are never latched,
> they're only ever used in wait-for-SIPI state.
> 
>>> The sender should set a flag as early as possible in the SIPI code so
>>> that it's clear that it was not received; and an extra SIPI is not a
>>> problem, it will be ignored anyway and will not cause trouble if
>>> there's a race.
>>>
>>> What is the reproducer for this?
>>
>> Hotplugging/unplugging cpus in a loop, especially if you oversubscribe
>> the guest, will get you there in 10-15 minutes.
>>
>> Typically (although I think not always) this is happening when OVMF if
>> trying to rendezvous and a processor is missing and is sent an extra SMI.
> 
> Can you go into more detail? I wasn't even aware that OVMF's SMM
> supported hotplug - on real hardware I think there's extra work from
> the BMC to coordinate all SMIs across both existing and hotplugged
> packages(*)


It's been supported by OVMF for a couple of years (in fact, IIRC you 
were part of at least initial conversations about this, at least for the 
unplug part).

During hotplug QEMU gathers all cpus in OVMF from (I think) 
ich9_apm_ctrl_changed() and they are all waited for in 
SmmCpuRendezvous()->SmmWaitForApArrival(). Occasionally it may so happen 
that the SMI from QEMU is not delivered to a processor that was *just* 
successfully hotplugged and so it is pinged again 
(https://github.com/tianocore/edk2/blob/fcfdbe29874320e9f876baa7afebc3fca8f4a7df/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c#L304). 


At the same time this processor is now being brought up by kernel and is 
being sent INIT-SIPI-SIPI. If these (or at least the SIPIs) arrive after 
the SMI reaches the processor then that processor is not going to have a 
good day.


> 
> What should happen is that SMIs are blocked on the new CPUs, so that
> only existing CPUs answer. These restore the 0x30000 segment to
> prepare for the SMI on the new CPUs, and send an INIT-SIPI to start
> the SMI on the new CPUs. Does OVMF do anything like that?
You mean this: 
https://github.com/tianocore/edk2/blob/fcfdbe29874320e9f876baa7afebc3fca8f4a7df/OvmfPkg/CpuHotplugSmm/Smbase.c#L272 
?


-boris