Received-SPF: pass (google.com: domain of linux-kernel+bounces-148484-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Wed, 17 Apr 2024 14:40:41 +0200
From: Igor Mammedov <imammedo@redhat.com>
To: boris.ostrovsky@oracle.com
Cc: Sean Christopherson <seanjc@google.com>, Paolo Bonzini
 <pbonzini@redhat.com>, kvm@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH] KVM/x86: Do not clear SIPI while in SMM
Message-ID: <20240417144041.1a493235@imammedo.users.ipa.redhat.com>
In-Reply-To: <77f30c15-9cae-46c2-ba2c-121712479b1c@oracle.com>
References: <20240416204729.2541743-1-boris.ostrovsky@oracle.com>
	<c7091688-8af5-4e70-b2d7-6d0a7134dbbe@redhat.com>
	<66cc2113-3417-42d0-bf47-d707816cbb53@oracle.com>
	<CABgObfZ-dFnWK46pyvuaO8TKEKC5pntqa1nXm-7Cwr0rpg5a3w@mail.gmail.com>
	<77fe7722-cbe9-4880-8096-e2c197c5b757@oracle.com>
	<Zh8G-AKzu0lvW2xb@google.com>
	<77f30c15-9cae-46c2-ba2c-121712479b1c@oracle.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Tue, 16 Apr 2024 19:37:09 -0400
boris.ostrovsky@oracle.com wrote:

> On 4/16/24 7:17 PM, Sean Christopherson wrote:
> > On Tue, Apr 16, 2024, boris.ostrovsky@oracle.com wrote: =20
> >> (Sorry, need to resend)
> >>
> >> On 4/16/24 6:03 PM, Paolo Bonzini wrote: =20
> >>> On Tue, Apr 16, 2024 at 10:57=E2=80=AFPM <boris.ostrovsky@oracle.com>=
 wrote: =20
> >>>> On 4/16/24 4:53 PM, Paolo Bonzini wrote: =20
> >>>>> On 4/16/24 22:47, Boris Ostrovsky wrote: =20
> >>>>>> Keeping the SIPI pending avoids this scenario. =20
> >>>>>
> >>>>> This is incorrect - it's yet another ugly legacy facet of x86, but =
we
> >>>>> have to live with it.  SIPI is discarded because the code is suppos=
ed
> >>>>> to retry it if needed ("INIT-SIPI-SIPI"). =20
> >>>>
> >>>> I couldn't find in the SDM/APM a definitive statement about whether =
SIPI
> >>>> is supposed to be dropped. =20
> >>>
> >>> I think the manual is pretty consistent that SIPIs are never latched,
> >>> they're only ever used in wait-for-SIPI state.
> >>> =20
> >>>>> The sender should set a flag as early as possible in the SIPI code =
so
> >>>>> that it's clear that it was not received; and an extra SIPI is not a
> >>>>> problem, it will be ignored anyway and will not cause trouble if
> >>>>> there's a race.
> >>>>>
> >>>>> What is the reproducer for this? =20
> >>>>
> >>>> Hotplugging/unplugging cpus in a loop, especially if you oversubscri=
be
> >>>> the guest, will get you there in 10-15 minutes.
> >>>>
> >>>> Typically (although I think not always) this is happening when OVMF =
if
> >>>> trying to rendezvous and a processor is missing and is sent an extra=
 SMI. =20
> >>>
> >>> Can you go into more detail? I wasn't even aware that OVMF's SMM
> >>> supported hotplug - on real hardware I think there's extra work from
> >>> the BMC to coordinate all SMIs across both existing and hotplugged
> >>> packages(*) =20
> >>
> >>
> >> It's been supported by OVMF for a couple of years (in fact, IIRC you w=
ere
> >> part of at least initial conversations about this, at least for the un=
plug
> >> part).
> >>
> >> During hotplug QEMU gathers all cpus in OVMF from (I think)
> >> ich9_apm_ctrl_changed() and they are all waited for in
> >> SmmCpuRendezvous()->SmmWaitForApArrival(). Occasionally it may so happ=
en
> >> that the SMI from QEMU is not delivered to a processor that was *just*
> >> successfully hotplugged and so it is pinged again (https://github.com/=
tianocore/edk2/blob/fcfdbe29874320e9f876baa7afebc3fca8f4a7df/UefiCpuPkg/PiS=
mmCpuDxeSmm/MpService.c#L304).
> >>
> >>
> >> At the same time this processor is now being brought up by kernel and =
is
> >> being sent INIT-SIPI-SIPI. If these (or at least the SIPIs) arrive aft=
er the
> >> SMI reaches the processor then that processor is not going to have a g=
ood
> >> day.=20

Do you use qemu/firmware combo that negotiated ICH9_LPC_SMI_F_CPU_HOTPLUG_B=
IT/
ICH9_LPC_SMI_F_CPU_HOT_UNPLUG_BIT features?

> >=20
> > It's specifically SIPI that's problematic.  INIT is blocked by SMM, but=
 latched,
> > and SMIs are blocked by WFS, but latched.  And AFAICT, KVM emulates all=
 of those
> > combinations correctly.
> >=20
> > Why is the SMI from QEMU not delivered?  That seems like the smoking gu=
n. =20
>=20
> I haven't actually traced this but it seems that what happens is that cv
> the newly-added processor is about to leave SMM and the count of in-SMM=20
> processors is decremented. At the same time, since the processor is=20
> still in SMM the QEMU's SMM is not taken.
>=20
> And so when the count is looked at again in SmmWaitForApArrival() one=20
> processor is missing.

Current QEMU CPU hotplug workflow with SMM enabled, should be following:

  1. OSPM gets list(N) of hotplugged cpus=20
  2. OSPM hands over control to firmware (SMM callback leading to SMI broad=
cast)
  3. Firmware at this point shall initialize all new CPUs (incl. relocating=
 SMBASE for new ones)
     it shall pull in all CPUs that are present at the moment
  4. Firmware returns control to OSPM
  5. OSPM sends Notify to the list(N) CPUs triggering INIT-SIPI-SIPI _only_=
 on
     those CPUs that it collected in step 1

above steps will repeat until all hotplugged CPUs are handled.

In nutshell INIT-SIPI-SIPI shall not be sent to a freshly hotplugged CPU
that OSPM haven't seen (1) yet _and_ firmware should have initialized (3).

CPUs enumerated at (3) at least shall include CPUs present at (1)
and may include newer CPU arrived in between (1-3).

CPUs collected at (1) shall all get SMM, if it doesn't happen
then hotplug workflow won't work as expected.
In which case we need to figure out why SMM is not delivered
or why firmware isn't waiting for hotplugged CPU.

>=20
> -boris
>=20