Received-SPF: pass (google.com: domain of linux-kernel+bounces-148640-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <cdbd1e4e-a5a3-4c3f-92e5-deee8d26280b@oracle.com>
Date: Wed, 17 Apr 2024 09:58:50 -0400
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] KVM/x86: Do not clear SIPI while in SMM
To: Igor Mammedov <imammedo@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>,
        Paolo Bonzini <pbonzini@redhat.com>, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org
References: <20240416204729.2541743-1-boris.ostrovsky@oracle.com>
 <c7091688-8af5-4e70-b2d7-6d0a7134dbbe@redhat.com>
 <66cc2113-3417-42d0-bf47-d707816cbb53@oracle.com>
 <CABgObfZ-dFnWK46pyvuaO8TKEKC5pntqa1nXm-7Cwr0rpg5a3w@mail.gmail.com>
 <77fe7722-cbe9-4880-8096-e2c197c5b757@oracle.com>
 <Zh8G-AKzu0lvW2xb@google.com>
 <77f30c15-9cae-46c2-ba2c-121712479b1c@oracle.com>
 <20240417144041.1a493235@imammedo.users.ipa.redhat.com>
Content-Language: en-US
From: boris.ostrovsky@oracle.com
In-Reply-To: <20240417144041.1a493235@imammedo.users.ipa.redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?d2daNXRXbWhTV2JIRzdrYmY0bnczcm5XMHFEdUFmazlDSEdHN3h1R0l1V3JG?=
 =?utf-8?B?ejZILy82Zk1xaDBTNDJBbTFNd3Y1Y0xialRXL1NNSWR0RjRqN0ZVeHhnT1dV?=
 =?utf-8?B?WURTWXJNVm45OG51eWdNblgxZlhhdloxei93c0Z1MjFIWnYxdzBxTm5QRjA0?=
 =?utf-8?B?ZmpNaEl2cVE4dXZMNWxvdUNQT081Q05yUFMwS0w5MnV3ZlBha29LbXdFL3JD?=
 =?utf-8?B?T3NVRlpBVXZJaEIyOUVwSFpRR3N2VkJOa2lsWnA4YnFvRXkwZGxEZndHcjFm?=
 =?utf-8?B?RzdMbHJPRDBTZi9SVkEzRFJBOWdwS29raDc4TFo3TU1ORC9xamxUN2RpQ0U5?=
 =?utf-8?B?L09SdjkvWmlncmd6aExwQlZDWHRBMS9tbWk2cWdDeDBnTm1OQ2J0NTFmOE5n?=
 =?utf-8?B?QWNING5EYVNhNGovK2JMVFJzN3NKY1BrR08yNkFJYUxibG5odWFiUHhFUkRP?=
 =?utf-8?B?NzBSUTZtYVlRa2lRL1JXbjZFdzFvRjJ4KytXblp6dHJuLzJXaEJ1RTB0bVMy?=
 =?utf-8?B?WTQzL3c3Uy9DendyMXQyV2dveEdueDRTbC9BYnZJVkV1YnlsL2VWbndJNlkr?=
 =?utf-8?B?Vy8zU09kcjN0YjByMGxJUDZFMHkzM2pyakFuYVZxSVhOcVpuVThid2Mwblhs?=
 =?utf-8?B?cE5wQmI5MFhDNFpHUENZbjF5anVKWk9ZQWJsNGp5ZTMrVWw5dFl5WE5rT0sz?=
 =?utf-8?B?SWEvVW8xS3UzelNQdnAvMjN2eVZlWjBZUEE3WmludzNTWXRTa0dCL0J6MGd1?=
 =?utf-8?B?Z2N3WW9Wc1RpQlUyZ3F2a0VqSklyMG9LM0c0RUl4VGRPQzFTbXlrMHZ6bnRn?=
 =?utf-8?B?NGhZQk9TTkVNUm1VRkNpL09Scm9lZmpDc09pTjZjZm5YZjNzRmpjMnNncU1n?=
 =?utf-8?B?a2VNQ0ZaTUJ2UXFzWUUycncrSnZrbkluWWJQczZuaFdCTmlCSXRCZE53SWVC?=
 =?utf-8?B?SGJ1M3hyM3U2aGVNVk9FVXJGZmoyVlZLL3dzWG9rSGUrOElHUVI4WWEyY3Rj?=
 =?utf-8?B?c2VVOGFGOWQvNUNtVVlMbWlJenRVa0FkSUo0TXdyaEhOZ2dPL1l0NjRrc3U4?=
 =?utf-8?B?SUIveUxhbm4yYzFrdHNvVEhWbVIzNlR0dCt3d05wU1YrZkNDMmpvVzJ1QmFv?=
 =?utf-8?B?WWh5OW1RYldIM1QyL0FFdVR3V0ZsRmkwWFFFenhzUDYwSGtNdS90c2k4SU13?=
 =?utf-8?B?aG4yUFFBUHQyNFd0bEdUdEdFV3BTdGxNb2loS091SXM0V2hKRktPNVlBblVH?=
 =?utf-8?B?a2p6bjJyMGZya01mT2g4QkJ5ZkREOVZiSnp0b09GaEFXQmFWOEhRQk9aSUJG?=
 =?utf-8?B?eVh2WGJjWGxOeXpyNk9oeWk2Nmh5a2JWaTZsU29BUjdUNmpSb0twbEpCK3A4?=
 =?utf-8?B?d2gvUVhGL1JIbEZveEVzWndKSlZYVkdEcmhkSFVCM2J2U1FEdWN3Rzd2ZU1p?=
 =?utf-8?B?aC9relRSVWQ3VHVhMDdvQTZnNEg3NnJHUmpKUXE0M2YzTzFzbEtmTU5wVjNG?=
 =?utf-8?B?ZVpEeXdGUTZYT3FmRmxKMGQyM3ZVQ2dnODVKQnhOMXc5R2Rxd1IrMTJleFJk?=
 =?utf-8?B?S2NtZ3RJVHBsczJ4d0ZvU0N3TFl5Tk16MTF6em1mdE5wYlpOQnVtQVZhY3g3?=
 =?utf-8?B?dG8wbXJjMnp5R3FQV1ZWeHNjeWV1aENTWFNqRFhOT1VWMXNTWm1kL0JRYmFx?=
 =?utf-8?B?Y3lhKzQ4U1lGaEtTQ3p2NFBybzNZQTc5dlJKakhUZnp3WklRMUNBZU8zMTJ2?=
 =?utf-8?B?cDNuYm1JT21FVnZpeTcwN0tZZm1IZWRoZUVBR1pjb05aQjkxKytSMGpOVm9u?=
 =?utf-8?B?SlVrNjVkNllyeUZva214QVFvTFdxcUNwYWtwNU5rQkRqd0dEQVdjYmJ4SEVJ?=
 =?utf-8?B?RHhVUGU4d2J4RWFsNjhJZklqT0xHSlhvNmh1bDVMdzVKMlJCeS9ZTlZISGxt?=
 =?utf-8?B?NUVCenRpY1Yva2RGSHd0eDE0VmQwaXRoRmFWTUhCZXdQU3pVbmJNYWh4elB1?=
 =?utf-8?B?RGlsZmpNL0xYejM5d2tvWS9FV1cyaWFCNjZKbDY4U28yUFFhbDhoYmc5Z2tK?=
 =?utf-8?B?cjRrOHVFMmZkMHl3bEgvR2hNMGhrcytIbnJmUTBXYjAzYlBzQmMyRy9zZlRX?=
 =?utf-8?B?MGhJMmR1WS8xL21ZbU9jQTRTUCtLMmlNZGEwcTBpNUJNTGsrSHpEV3hZTktt?=
 =?utf-8?B?ZEE9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	YqzwI6Ie9lHlrm8A4PVOH0N8PJZrIWegnIsQ6YDi3dXGcqHSWGNhU3Llw+119QZIBo8KZuLAfTlH4Wltnq3ovvVxSCWz/BTNf4g+MG7zciUwtwF9S+duDpQkC0PaQlJhVziFRrR/nNPHqqP3UpBvJsShLyKjv0QCUwQ0uH1bcrUMgfcBNSum6m4Va2DK4P68vKlu+wmrgVhzL5SwZ7c8naBOscWJNVskesXSJvg+QaO2Pap006Jg1CDDKZjd2dYcJ74lGopgWXN26DshQF8TtJRPH763h2X5IC2B7DGxluN0VHpi0vkZMhyevMhVwnJN6NAQHnYsJKroPUaWDrSZLeka8J2ejtw+GxrnF5/PMju+Dq9LlnHffTEhrwYHsKxBuOhr6LIDe+vpfWn9FoGhBI5Bq5kNG/1sIOYoqDTLGKTADwjrlq4BmZ/EgBO3AyK4fFN10PhtVedJG6pafziN/nVVK0HGFSUtigyb5zgJDUV8sLxeT9Q9UFKHeG9Vmc3stx0UkHM/EYhY2lKNDJjcJqysdLk7fFMbtuQiPTe94P+mcR+DliIeAHJg3zpFgFJNxa3aK1kwq/E9JRvZokVzOMC9ljAuRI26J/vJPqhfOm8=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 978ff25f-3257-426a-f846-08dc5ee684c8
X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB5009.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Apr 2024 13:58:54.4730
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /ZYI3tkNZkG/IZzw6u3QTvyuKOTzExWNrLGvDsIduEIZAZCiB+tts/mIugG/B5IVL7CwSA1EY8iutiAt/8tnJFgr80cJTcxHWCtTBEQKcOQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6583
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-04-17_11,2024-04-16_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 mlxlogscore=999
 suspectscore=0 adultscore=0 phishscore=0 bulkscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2404010000
 definitions=main-2404170096
X-Proofpoint-ORIG-GUID: _p-l6ZTD_nkunlDyumZ3h2tyDXXIVSLG
X-Proofpoint-GUID: _p-l6ZTD_nkunlDyumZ3h2tyDXXIVSLG


On 4/17/24 8:40 AM, Igor Mammedov wrote:
> On Tue, 16 Apr 2024 19:37:09 -0400
> boris.ostrovsky@oracle.com wrote:
> 
>> On 4/16/24 7:17 PM, Sean Christopherson wrote:
>>> On Tue, Apr 16, 2024, boris.ostrovsky@oracle.com wrote:
>>>> (Sorry, need to resend)
>>>>
>>>> On 4/16/24 6:03 PM, Paolo Bonzini wrote:
>>>>> On Tue, Apr 16, 2024 at 10:57 PM <boris.ostrovsky@oracle.com> wrote:
>>>>>> On 4/16/24 4:53 PM, Paolo Bonzini wrote:
>>>>>>> On 4/16/24 22:47, Boris Ostrovsky wrote:
>>>>>>>> Keeping the SIPI pending avoids this scenario.
>>>>>>>
>>>>>>> This is incorrect - it's yet another ugly legacy facet of x86, but we
>>>>>>> have to live with it.  SIPI is discarded because the code is supposed
>>>>>>> to retry it if needed ("INIT-SIPI-SIPI").
>>>>>>
>>>>>> I couldn't find in the SDM/APM a definitive statement about whether SIPI
>>>>>> is supposed to be dropped.
>>>>>
>>>>> I think the manual is pretty consistent that SIPIs are never latched,
>>>>> they're only ever used in wait-for-SIPI state.
>>>>>   
>>>>>>> The sender should set a flag as early as possible in the SIPI code so
>>>>>>> that it's clear that it was not received; and an extra SIPI is not a
>>>>>>> problem, it will be ignored anyway and will not cause trouble if
>>>>>>> there's a race.
>>>>>>>
>>>>>>> What is the reproducer for this?
>>>>>>
>>>>>> Hotplugging/unplugging cpus in a loop, especially if you oversubscribe
>>>>>> the guest, will get you there in 10-15 minutes.
>>>>>>
>>>>>> Typically (although I think not always) this is happening when OVMF if
>>>>>> trying to rendezvous and a processor is missing and is sent an extra SMI.
>>>>>
>>>>> Can you go into more detail? I wasn't even aware that OVMF's SMM
>>>>> supported hotplug - on real hardware I think there's extra work from
>>>>> the BMC to coordinate all SMIs across both existing and hotplugged
>>>>> packages(*)
>>>>
>>>>
>>>> It's been supported by OVMF for a couple of years (in fact, IIRC you were
>>>> part of at least initial conversations about this, at least for the unplug
>>>> part).
>>>>
>>>> During hotplug QEMU gathers all cpus in OVMF from (I think)
>>>> ich9_apm_ctrl_changed() and they are all waited for in
>>>> SmmCpuRendezvous()->SmmWaitForApArrival(). Occasionally it may so happen
>>>> that the SMI from QEMU is not delivered to a processor that was *just*
>>>> successfully hotplugged and so it is pinged again (https://github.com/tianocore/edk2/blob/fcfdbe29874320e9f876baa7afebc3fca8f4a7df/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c#L304).
>>>>
>>>>
>>>> At the same time this processor is now being brought up by kernel and is
>>>> being sent INIT-SIPI-SIPI. If these (or at least the SIPIs) arrive after the
>>>> SMI reaches the processor then that processor is not going to have a good
>>>> day.
> 
> Do you use qemu/firmware combo that negotiated ICH9_LPC_SMI_F_CPU_HOTPLUG_BIT/
> ICH9_LPC_SMI_F_CPU_HOT_UNPLUG_BIT features?

Yes.

> 
>>>
>>> It's specifically SIPI that's problematic.  INIT is blocked by SMM, but latched,
>>> and SMIs are blocked by WFS, but latched.  And AFAICT, KVM emulates all of those
>>> combinations correctly.
>>>
>>> Why is the SMI from QEMU not delivered?  That seems like the smoking gun.
>>
>> I haven't actually traced this but it seems that what happens is that cv
>> the newly-added processor is about to leave SMM and the count of in-SMM
>> processors is decremented. At the same time, since the processor is
>> still in SMM the QEMU's SMM is not taken.
>>
>> And so when the count is looked at again in SmmWaitForApArrival() one
>> processor is missing.
> 
> Current QEMU CPU hotplug workflow with SMM enabled, should be following:
> 
>    1. OSPM gets list(N) of hotplugged cpus
>    2. OSPM hands over control to firmware (SMM callback leading to SMI broadcast)
>    3. Firmware at this point shall initialize all new CPUs (incl. relocating SMBASE for new ones)
>       it shall pull in all CPUs that are present at the moment
>    4. Firmware returns control to OSPM
>    5. OSPM sends Notify to the list(N) CPUs triggering INIT-SIPI-SIPI _only_ on
>       those CPUs that it collected in step 1
> 
> above steps will repeat until all hotplugged CPUs are handled.
> 
> In nutshell INIT-SIPI-SIPI shall not be sent to a freshly hotplugged CPU
> that OSPM haven't seen (1) yet _and_ firmware should have initialized (3).
> 
> CPUs enumerated at (3) at least shall include CPUs present at (1)
> and may include newer CPU arrived in between (1-3).
> 
> CPUs collected at (1) shall all get SMM, if it doesn't happen
> then hotplug workflow won't work as expected.
> In which case we need to figure out why SMM is not delivered
> or why firmware isn't waiting for hotplugged CPU.

I noticed that I was using a few months old qemu bits and now I am 
having trouble reproducing this on latest bits. Let me see if I can get 
this to fail with latest first and then try to trace why the processor 
is in this unexpected state.

-boris