Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 22 Mar 2023 16:42:59 -0700
In-Reply-To: <c9abb017-0088-b96f-d19a-b25e54ad150d@citrix.com>
Mime-Version: 1.0
References: <20230307023946.14516-1-xin3.li@intel.com> <20230307023946.14516-35-xin3.li@intel.com>
 <ZBs/sSJwr7zdOUsE@google.com> <c9abb017-0088-b96f-d19a-b25e54ad150d@citrix.com>
Message-ID: <ZBuSg+3jJpjA5S5P@google.com>
Subject: Re: [PATCH v5 34/34] KVM: x86/vmx: execute "int $2" to handle NMI in
 NMI caused VM exits when FRED is enabled
From:   Sean Christopherson <seanjc@google.com>
To:     andrew.cooper3@citrix.com
Cc:     Xin Li <xin3.li@intel.com>, linux-kernel@vger.kernel.org,
        x86@kernel.org, kvm@vger.kernel.org, tglx@linutronix.de,
        mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com,
        hpa@zytor.com, peterz@infradead.org, pbonzini@redhat.com,
        ravi.v.shankar@intel.com
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Wed, Mar 22, 2023, andrew.cooper3@citrix.com wrote:
> On 22/03/2023 5:49 pm, Sean Christopherson wrote:
> > On Mon, Mar 06, 2023, Xin Li wrote:
> >> Execute "int $2" to handle NMI in NMI caused VM exits when FRED is ena=
bled.
> >>
> >> Like IRET for IDT, ERETS/ERETU are required to end the NMI handler for=
 FRED
> >> to unblock NMI ASAP (w/ bit 28 of CS set).
> > That's "CS" on the stack correct?  Is bit 28 set manually by software, =
or is it
> > set automatically by hardware?  If it's set by hardware, does "int $2" =
actually
> > set the bit since it's not a real NMI?
>=20
> int $2 had better not set it...=EF=BF=BD This is the piece of state that =
is
> intended to cause everything which isn't a real NMI to nest properly
> inside a real NMI.
>=20
> It is supposed to be set on delivery of an NMI, and act as the trigger
> for ERET{U,S} to drop the latch.
>=20
> Software is can set it manually in a FRED-frame in order to explicitly
> unblock NMIs.

Ah, found this in patch 19.  That hunk really belongs in this patch, becaus=
e this
patch is full of magic without that information.

+       /*
+        * VM exits induced by NMIs keep NMI blocked, and we do
+        * "int $2" to reinject the NMI w/ NMI kept being blocked.
+        * However "int $2" doesn't set the nmi bit in the FRED
+        * stack frame, so we explicitly set it to make sure a
+        * later ERETS will unblock NMI immediately.
+        */
+       regs->nmi =3D 1;

Organization aside, this seems to defeat the purpose of _not_ unconditional=
ly
unmasking NMIs on ERET since the kernel assumes any random "int $2" is comi=
ng from
KVM after an NMI VM-Exit.

Eww, and "int $2" doesn't even go directly to fred_exc_nmi(), it trampoline=
s
through fred_sw_interrupt_kernel() first.  Looks like "int $2" from userspa=
ce gets
routed to a #GP, so at least that bit is handled.

I'm not dead set against the proposed approach, but IMO it's not obviously =
better
than a bit of assembly to have a more direct call into the NMI handler.