Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33;
From:   "Li, Xin3" <xin3.li@intel.com>
To:     "Gao, Chao" <chao.gao@intel.com>
CC:     "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-hyperv@vger.kernel.org" <linux-hyperv@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "seanjc@google.com" <seanjc@google.com>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "kys@microsoft.com" <kys@microsoft.com>,
        "haiyangz@microsoft.com" <haiyangz@microsoft.com>,
        "wei.liu@kernel.org" <wei.liu@kernel.org>,
        "Cui, Dexuan" <decui@microsoft.com>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "bp@alien8.de" <bp@alien8.de>,
        "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
        "x86@kernel.org" <x86@kernel.org>, "hpa@zytor.com" <hpa@zytor.com>,
        "vkuznets@redhat.com" <vkuznets@redhat.com>,
        "peterz@infradead.org" <peterz@infradead.org>,
        "Shankar, Ravi V" <ravi.v.shankar@intel.com>
Subject: RE: [PATCH v1 13/23] KVM: VMX: Handle VMX nested exception for FRED
Thread-Topic: [PATCH v1 13/23] KVM: VMX: Handle VMX nested exception for FRED
Thread-Index: AQHaEnYXOQlvFLhpWkG4lt/jksLCm7B5dqYAgAFC3tCAIVOmcIABoIUAgAAStHCAAQ43gIABYtvg
Date:   Fri, 8 Dec 2023 23:48:37 +0000
Message-ID: <SA1PR11MB6734FFE3D3500BFE4AF407DDA88AA@SA1PR11MB6734.namprd11.prod.outlook.com>
References: <20231108183003.5981-1-xin3.li@intel.com>
 <20231108183003.5981-14-xin3.li@intel.com> <ZVMkVmBPVfaMjDTL@chao-email>
 <SA1PR11MB67348D3637C2BC6B107C5CCAA8B1A@SA1PR11MB6734.namprd11.prod.outlook.com>
 <SA1PR11MB6734EFF17E15C68AAD12A227A884A@SA1PR11MB6734.namprd11.prod.outlook.com>
 <ZXGFcuwwmkBlqq5s@chao-email>
 <SA1PR11MB67342A44E1C2E17E127AA44AA88BA@SA1PR11MB6734.namprd11.prod.outlook.com>
 <ZXJ3z0yCqnHm52sC@chao-email>
In-Reply-To: <ZXJ3z0yCqnHm52sC@chao-email>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?SqWlLhGr0ehKIw+fIeqmHonP+AxpxjREUUsVNmQBOuzS6l1bGV139U1OQq9l?=
 =?us-ascii?Q?BCsv0ggAU2nrcyS8EO7hj62Kk9R1AZeDcSQ5clanfynr6WhG1ZKbr7WvG4K8?=
 =?us-ascii?Q?H9Qo3OPFkvYjtS/+3+KhyK2bt+T0XMEAYNPJiSqQcsj9Ndo2/83nDM0blqyV?=
 =?us-ascii?Q?2uANXSL39M3L4ltbjXf0Ve2Fg03cbsA1tzEeCwxwJCGe+5A+EZ+oBPLTD4+7?=
 =?us-ascii?Q?IEhfdDI7H1iN/8ZlaqqcNcbbtBvy9qlto+rJHTskyqkMu8lZdgHc0Dl+fx/9?=
 =?us-ascii?Q?VaOrpefv5lK8jsr++PZBrb+ziZhEyFwO4pHYe8Pdn6cQfjXq1p4JtT6YtNbP?=
 =?us-ascii?Q?8n5RTZwGwfu87M5KXImAIFR0C+BoFnP7nLxh3fqsIO7mnOVsO5vLvu/YAgou?=
 =?us-ascii?Q?u0kpWWukcjUvO75TP0JawjG2jRbxB3hbZGSWY6UbKU/IT+eg4Jn1wPhndtIr?=
 =?us-ascii?Q?1rsFOs5Sf4Qb6T9Lo3uXJX/z9At0rLAnvQZBZYS7yVZqTAbIOYd222cDt7or?=
 =?us-ascii?Q?14w6tTOoUo3Ge6YJ9Fz6GRfACQvEHIeYNqjDeop7bc48Xtey1gFSl6lTm8fC?=
 =?us-ascii?Q?8XVd9L+qJNyB0hiqFpdXattuD9ftE03G6NoVB24c8XnhnX/yVOdpxA0sN/LQ?=
 =?us-ascii?Q?TzyeLnue41wP2IbLDR51kAVe4ZDZYQvpIEnjnfPTkBxBPQW6Er5Ly3bVNj/E?=
 =?us-ascii?Q?mlyZPq+pBcuvxc7LHH9IdbruNOpfWlx2tPamfYSJYv8dNvCOPry0WVuGGZUT?=
 =?us-ascii?Q?35M7J0AduTkArZeiZT4WdF2UuZqY2f9c4Ry3sCQPHzq45uHvYvTTL2bLleRa?=
 =?us-ascii?Q?jIYKZz5hkORgCAUpSrNeSlraqYAq95zqUPN8wiQsX7Nr0BkW7ZESCO5omDFb?=
 =?us-ascii?Q?AqQPa7mUgiGLEQ7Hnmxm6iXNIFDw2ZEuvg1kPopkPpWpz5oEY7qkIgxrYFjL?=
 =?us-ascii?Q?/dbJJLJjk7x75rJsal70gLmBxjDsfGq4B3psKTQo01G4K11OfYvYJPlft3cx?=
 =?us-ascii?Q?mmDsi3R1mjs0sbZqWrO8pKGIMqF1OI9ctwXwhdUOnpR8E5BDeGJJEnuKgxVQ?=
 =?us-ascii?Q?/j/DxC2N9NXImGgWmo0UvuubSY1GjnlWQ8eBE2+Rt052K7UlZfGRor0DlcKz?=
 =?us-ascii?Q?EHZhkCEwcDaxAKGnxuJk97vkRiS9S4Lza5vMHg52QeMBv6JBeS1uEotHkAdS?=
 =?us-ascii?Q?4gNIw2mYcgfaQtfpJvQ/iVFX0klpXp8kE/KSWLF5hDz594IbFnbmLm2pa89L?=
 =?us-ascii?Q?LsXPxbSXMpdLL7B4Hoh9Rwfi4lQWXLM6w9rR+agVl+uS1zHRIS/tcw8AQaLf?=
 =?us-ascii?Q?9fEu8fOKvsE4dkknovht2V4MkWUoi40pWJR5ROFZDnwjW9xi609Fq5Tjn0Ws?=
 =?us-ascii?Q?zCKP0RKEUf7JM5AF9hevZveQzCGR0ogRlNOofEOEw7yVS31TpRhKio6WEuvA?=
 =?us-ascii?Q?8SY23SC2P2pXTTV+23ZqsjYMPnNvGc5UGPAVHBO/EPb0Iip6GYgOJ5yupEFb?=
 =?us-ascii?Q?OG2ECcplOrLjB8VamFlY5qwCuzVV57RTsuZGtxU4rNMMzGWOR1KzSc6kmCVi?=
 =?us-ascii?Q?tB6x8CKHXcYY2d1A5IU=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SA1PR11MB6734.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f59a9df9-0570-4e8f-e8b1-08dbf84832c1
X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Dec 2023 23:48:37.5801
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: I59jxCVWQgJeaAOmZLmBEII1gagSyIbMZMt2RBqHksvZoH8lhLngn4nMbfVURc1Ev7HUJdHzr6JNXX37ysb5oQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB6622
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 08 Dec 2023 15:48:58 -0800 (PST)

> >> >> > Exiting-event identification can also have bit 13 set, indicating=
 a
> >> >> > nested exception encountered and caused VM-exit. when reinjecting=
 the
> >> >> > exception to guests, kvm needs to set the "nested" bit, right? I
> >> >> > suspect some changes to e.g., handle_exception_nmi() are needed.
> >> >>
> >> >> The current patch relies on kvm_multiple_exception() to do that.  B=
ut TBH,
> I'm
> >> >> not sure it can recognize all nested cases.  I probably should revi=
sit it.
> >> >
> >> >So the conclusion is that kvm_multiple_exception() is smart enough, a=
nd
> >> >a VMM doesn't have to check bit 13 of the Exiting-event identificatio=
n.
> >> >
> >> >In FRED spec 5.0, section 9.2 - New VMX Feature: VMX Nested-Exception
> >> >Support, there is a statement at the end of Exiting-event identificat=
ion:
> >> >
> >> >(The value of this bit is always identical to that of the valid bit o=
f
> >> >the original-event identification field.)
> >> >
> >> >It means that even w/o VMX Nested-Exception support, a VMM already
> knows
> >> >if an exception is a nested exception encountered during delivery of
> >> >another event in an exception caused VM exit (exit reason 0).  This i=
s
> >> >done in KVM through reading IDT_VECTORING_INFO_FIELD and calling
> >> >vmx_complete_interrupts() immediately after VM exits.
> >> >
> >> >vmx_complete_interrupts() simply queues the original exception if the=
re is
> >> >one, and later the nested exception causing the VM exit could be canc=
elled
> >> >if it is a shadow page fault.  However if the shadow page fault is ca=
used
> >> >by a guest page fault, KVM injects it as a nested exception to have g=
uest
> >> >fix its page table.
> >> >
> >> >I will add comments about this background in the next iteration.
> >>
> >> is it possible that the CPU encounters an exception and causes VM-exit=
 during
> >> injecting an __interrupt__? in this case, no __exception__ will be (re=
-)queued
> >> by vmx_complete_interrupts().
> >
> >I guess the following case is what you're suggesting:
> >KVM injects an external interrupt after shadow page tables are nuked.
> >
> >vmx_complete_interrupts() are called after each VM exit to clear both
> >interrupt and exception queues, which means it always pushes the
> >deepest event if there is an original event.  In the above case, the
> >original event is the external interrupt KVM just tried to inject.
>=20
> in my understanding, your point is:
> 1. if bit 13 of the Exiting-event identification is set. the original-eve=
nt
> identification field should be valid.
> 2. vmx_complete_interrupts() is done immediately after VM exits and reads
> original-event identification and reinjects the event there.
> 3. if KVM injects the exception in exiting-event identification
> to guest, KVM doesn't need to read the bit 13 because kvm_multiple_except=
ion()
> is "smart enough" and recognize the exception as nested-exception because=
 if
> bit 13 is 1, one exception must has been queued in #2.
>=20
> my question is:
> what if the event in original-event identification is an interrupt e.g.,
> external interrupt or NMI, rather than exception.  vmx_complete_interrupt=
s()
> won't queue an exception, then how can KVM or kvm_multiple_exception()
> know the
> exception that caused VM-exit is an nested exception w/o reading bit 13 o=
f the
> Exiting-event identification?

The good news is that vmx_complete_interrupts() still queues the event
even it's not a hardware exception.  It's just that kvm_multiple_exception(=
)
doesn't check if there is an original interrupt or NMI because IDT event
delivery doesn't care such a case.

I think your point is more of that we should check it when FRED is enabled
for a guest.  Yes, architecturally we should do it.

What I want to emphasize is that bit 13 of the exiting-event identification
is set to the valid bit of the original-event identification, they are
logically the same thing when FRED is enabled.  It doens't matter which one
a VMM reads and uses.  But a VMM doesn't need to differentiate FRED and IDT
if it reads the info from original-event identification.