Received-SPF: pass (google.com: domain of linux-kernel+bounces-147746-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Tue, 16 Apr 2024 16:17:12 -0700
In-Reply-To: <77fe7722-cbe9-4880-8096-e2c197c5b757@oracle.com>
Precedence: bulk
Mime-Version: 1.0
References: <20240416204729.2541743-1-boris.ostrovsky@oracle.com>
 <c7091688-8af5-4e70-b2d7-6d0a7134dbbe@redhat.com> <66cc2113-3417-42d0-bf47-d707816cbb53@oracle.com>
 <CABgObfZ-dFnWK46pyvuaO8TKEKC5pntqa1nXm-7Cwr0rpg5a3w@mail.gmail.com> <77fe7722-cbe9-4880-8096-e2c197c5b757@oracle.com>
Message-ID: <Zh8G-AKzu0lvW2xb@google.com>
Subject: Re: [PATCH] KVM/x86: Do not clear SIPI while in SMM
From: Sean Christopherson <seanjc@google.com>
To: boris.ostrovsky@oracle.com
Cc: Paolo Bonzini <pbonzini@redhat.com>, kvm@vger.kernel.org, linux-kernel@vger.kernel.org
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

On Tue, Apr 16, 2024, boris.ostrovsky@oracle.com wrote:
> (Sorry, need to resend)
>=20
> On 4/16/24 6:03 PM, Paolo Bonzini wrote:
> > On Tue, Apr 16, 2024 at 10:57=E2=80=AFPM <boris.ostrovsky@oracle.com> w=
rote:
> > > On 4/16/24 4:53 PM, Paolo Bonzini wrote:
> > > > On 4/16/24 22:47, Boris Ostrovsky wrote:
> > > > > Keeping the SIPI pending avoids this scenario.
> > > >=20
> > > > This is incorrect - it's yet another ugly legacy facet of x86, but =
we
> > > > have to live with it.  SIPI is discarded because the code is suppos=
ed
> > > > to retry it if needed ("INIT-SIPI-SIPI").
> > >=20
> > > I couldn't find in the SDM/APM a definitive statement about whether S=
IPI
> > > is supposed to be dropped.
> >=20
> > I think the manual is pretty consistent that SIPIs are never latched,
> > they're only ever used in wait-for-SIPI state.
> >=20
> > > > The sender should set a flag as early as possible in the SIPI code =
so
> > > > that it's clear that it was not received; and an extra SIPI is not =
a
> > > > problem, it will be ignored anyway and will not cause trouble if
> > > > there's a race.
> > > >=20
> > > > What is the reproducer for this?
> > >=20
> > > Hotplugging/unplugging cpus in a loop, especially if you oversubscrib=
e
> > > the guest, will get you there in 10-15 minutes.
> > >=20
> > > Typically (although I think not always) this is happening when OVMF i=
f
> > > trying to rendezvous and a processor is missing and is sent an extra =
SMI.
> >=20
> > Can you go into more detail? I wasn't even aware that OVMF's SMM
> > supported hotplug - on real hardware I think there's extra work from
> > the BMC to coordinate all SMIs across both existing and hotplugged
> > packages(*)
>=20
>=20
> It's been supported by OVMF for a couple of years (in fact, IIRC you were
> part of at least initial conversations about this, at least for the unplu=
g
> part).
>=20
> During hotplug QEMU gathers all cpus in OVMF from (I think)
> ich9_apm_ctrl_changed() and they are all waited for in
> SmmCpuRendezvous()->SmmWaitForApArrival(). Occasionally it may so happen
> that the SMI from QEMU is not delivered to a processor that was *just*
> successfully hotplugged and so it is pinged again (https://github.com/tia=
nocore/edk2/blob/fcfdbe29874320e9f876baa7afebc3fca8f4a7df/UefiCpuPkg/PiSmmC=
puDxeSmm/MpService.c#L304).
>=20
>=20
> At the same time this processor is now being brought up by kernel and is
> being sent INIT-SIPI-SIPI. If these (or at least the SIPIs) arrive after =
the
> SMI reaches the processor then that processor is not going to have a good
> day.

It's specifically SIPI that's problematic.  INIT is blocked by SMM, but lat=
ched,
and SMIs are blocked by WFS, but latched.  And AFAICT, KVM emulates all of =
those
combinations correctly.

Why is the SMI from QEMU not delivered?  That seems like the smoking gun.