Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Li, Xin3" <xin3.li@intel.com>
To:     "Christopherson,, Sean" <seanjc@google.com>
CC:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "x86@kernel.org" <x86@kernel.org>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "bp@alien8.de" <bp@alien8.de>,
        "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
        "hpa@zytor.com" <hpa@zytor.com>,
        "peterz@infradead.org" <peterz@infradead.org>,
        "andrew.cooper3@citrix.com" <andrew.cooper3@citrix.com>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "Shankar, Ravi V" <ravi.v.shankar@intel.com>,
        "jiangshanlai@gmail.com" <jiangshanlai@gmail.com>,
        "Kang, Shan" <shan.kang@intel.com>
Subject: RE: [PATCH v8 33/33] KVM: x86/vmx: refactor VMX_DO_EVENT_IRQOFF to
 generate FRED stack frames
Thread-Topic: [PATCH v8 33/33] KVM: x86/vmx: refactor VMX_DO_EVENT_IRQOFF to
 generate FRED stack frames
Thread-Index: AQHZa4iiYR92PWeRCEi6WbUcy97+Bq8lFcEAgALhuRA=
Date:   Wed, 12 Apr 2023 18:26:17 +0000
Message-ID: <SA1PR11MB6734897C0BC0FC7481333E29A89B9@SA1PR11MB6734.namprd11.prod.outlook.com>
References: <20230410081438.1750-1-xin3.li@intel.com>
 <20230410081438.1750-34-xin3.li@intel.com> <ZDSEjhGV9D90J6Bx@google.com>
In-Reply-To: <ZDSEjhGV9D90J6Bx@google.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?+guFN0IGHLxJFWcoQcRl08oyKF6Rvg9RFVqq5SRwy/irhG88ldQzyQZ0So0B?=
 =?us-ascii?Q?ng7Ay8ZbqqZOBde+ws/puaNYXX2LrdcoDlT7E9b7GzVAC5ZVxsR8NywjECvo?=
 =?us-ascii?Q?ytWJzOsCe9pxFbnS9Rgk9gHgy59k8tT2Nz8u69pjHj4ErnPl0Qv5kB1hAYWC?=
 =?us-ascii?Q?W7Jp0hYdwhKG9RgfQBvj1tKT2cgOhJoMBq9Q6p00xE740aC954XF0LKsav/L?=
 =?us-ascii?Q?wRjf+5zd0pyppngxi/FYruW2A0uAsAV+O1/4fPI8vRA3ArxEPoHB0de2sypm?=
 =?us-ascii?Q?ANO1D0cvZBh5hCALM8a2MX6n0KOGxfJwhL5u/3mfiJVYx8pKnO1ZtBU7ubc+?=
 =?us-ascii?Q?QgFC4Vjz594wH0MgWopO8PrBLX2AlsSNoWRVl5eLyjDW1MDIRiLJr0waeTFR?=
 =?us-ascii?Q?wPGYc4s5BvDOYloeZ6TR1tIm2oXTRDd1EpkwqgwQuaDXafPD+dlj1GzGp0ak?=
 =?us-ascii?Q?XVt8jM0+3+e8IIjFK7TqSnXPeStB6bEIhVcZYP1IBl35jMn06iOitiL/MbSx?=
 =?us-ascii?Q?exPQ0B9CeBCJSZ0jejhupYmZMyEWoHqRT+dNfpHLeHS8JbkrLnzhDp44NGfI?=
 =?us-ascii?Q?A94NzpkdTh5fZZJA1kwt+kXNSSovnWjPLVnvGQ10NSxqDaKpknwiosWMLR0B?=
 =?us-ascii?Q?+wU4iUIaXETKln6TH9hjfwPpVrV1dwTFsWphzALl+HR0WhFIadpq8y4VHN0D?=
 =?us-ascii?Q?rxNrUMZNtWiPeRcbK2GPGFbCQtiT7X5A2dUrLSsmRWtMwZiaU20DhOuyVWUu?=
 =?us-ascii?Q?wHI0zfLJ1788bNEmOV0JHj2zWkoZy8MNGGV1PeXzudKC6O/2kZWJfPGb2aoE?=
 =?us-ascii?Q?phSZKulT/0uWV3n8F2Y/cBoFeJoyXSI/u9jN8qsFiN/Dy6ymezKRwnDp3vXj?=
 =?us-ascii?Q?ESRirHrq6mQ6sf+AsIlLd7zUwAqhCOOJ50djmEeSIccudaVl2hx3ZPb3nnW9?=
 =?us-ascii?Q?zEQnTnAPbsLFN6bHWC42gfuqkrbCIhAIXi5zJAP/bPE5k68TqrJPMusXwK04?=
 =?us-ascii?Q?rnPipERWOZbM10HiTsl4NV8JnL/oqVbceSXOCePHMpm4ct7wE3g3TwikGIo7?=
 =?us-ascii?Q?7Tf+q1UblP9WmWK7lLv/O7XjrBSm7AgYmbT0i0azFA9jvEhJ1qPuPxZqPwyq?=
 =?us-ascii?Q?pZ5tkOIQPANurA2kONaQ8SwLNzXfO9RP+LNOWacaK+8VhNqeB32rHfHDyzKl?=
 =?us-ascii?Q?uLaTLktktdiO8101FsrNJsP9xO62qhx8Q8plEUJvnm5n3cIt2j6Hj3YfFepc?=
 =?us-ascii?Q?7tTvKd0YW1TKwlq+8msWw1+Sc74MzQtRb1ejAz8EG5Q12ZI0zfve7db3QUFK?=
 =?us-ascii?Q?mSSL8Jp3VyK4LuD98joTftYElv1VvVyMtXDxtgMvKaFmDfj++fOkg1zQek2j?=
 =?us-ascii?Q?PF1UG69+dyzXj0z6W5DS7nqaaScGuIVmyaPFsBN3sDUbVYhTDaWJYhOlL3vT?=
 =?us-ascii?Q?ytrjs1l5gmiXFL61XudeQzZzNW/o+kYdKa3+qgFoJhWfuT5dwnMG97adF9Qb?=
 =?us-ascii?Q?0VeXNpk4JIBYUhBoTzNjsIdmZ7T7t1mgxys2DTcekFE4jaRWm/mfLvrkGGF6?=
 =?us-ascii?Q?4HtT3RrKG8C8M+nXWys=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SA1PR11MB6734.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e5701316-e41a-4739-b772-08db3b83683a
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Apr 2023 18:26:17.8136
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: U9Vpmh2gAwvaoHchzU0KfF6RX9wmXsBNwvLHR8ow7dEr0TEOAg5wjA1tjiPP2hlge2W1iiromkpgkMoxqK6YuA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB5817
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> And then this is equally gross.  Rather than funnel FRED+legacy into a si=
ngle
> function only to split them back out, just route FRED into its own asm su=
broutine.
> The common bits are basically the creation/destruction of the stack frame=
 and
> the CALL itself, i.e. the truly interesting bits are what's different.

I try to catch up with you but am still confused.

Because a FRED stack frame always contains an error code pushed after RIP,
the FRED entry code doesn't push any error code.

Thus I introduced a trampoline code, which is called to have the return
instruction address pushed first. Then the trampoline code pushes an error
code (0 for both IRQ and NMI) and jumps to fred_entrypoint_kernel() for NMI
handling or calls external_interrupt() for IRQ handling.

The return RIP is used to return from fred_entrypoint_kernel(), but not
external_interrupt().


> Pretty much all of the #ifdeffery goes away, the helpers just need #ifdef=
s to play
> nice with CONFIG_X86_FRED=3Dn.  E.g. something like the below as a starti=
ng point
> (it most definitely doesn't compile, and most definitely isn't 100% corre=
ct).
>=20
> ---
>  arch/x86/kvm/vmx/vmenter.S | 72
> ++++++++++++++++++++++++++++++++++++++
>  arch/x86/kvm/vmx/vmx.c     | 19 ++++++++--
>  2 files changed, 88 insertions(+), 3 deletions(-)
>=20
> diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S inde=
x
> 631fd7da2bc3..a6929c78e038 100644
> --- a/arch/x86/kvm/vmx/vmenter.S
> +++ b/arch/x86/kvm/vmx/vmenter.S
> @@ -2,12 +2,14 @@
>  #include <linux/linkage.h>
>  #include <asm/asm.h>
>  #include <asm/bitsperlong.h>
> +#include <asm/fred.h>
>  #include <asm/kvm_vcpu_regs.h>
>  #include <asm/nospec-branch.h>
>  #include <asm/percpu.h>
>  #include <asm/segment.h>
>  #include "kvm-asm-offsets.h"
>  #include "run_flags.h"
> +#include "../../entry/calling.h"
>=20
>  #define WORD_SIZE (BITS_PER_LONG / 8)
>=20
> @@ -31,6 +33,62 @@
>  #define VCPU_R15	__VCPU_REGS_R15 * WORD_SIZE
>  #endif
>=20
> +#ifdef CONFIG_X86_FRED
> +.macro VMX_DO_FRED_EVENT_IRQOFF call_target cs_val
> +	/*
> +	 * Unconditionally create a stack frame, getting the correct RSP on the
> +	 * stack (for x86-64) would take two instructions anyways, and RBP can
> +	 * be used to restore RSP to make objtool happy (see below).
> +	 */
> +	push %_ASM_BP
> +	mov %_ASM_SP, %_ASM_BP
> +
> +	/*
> +	 * Don't check the FRED stack level, the call stack leading to this
> +	 * helper is effectively constant and shallow (relatively speaking).
> +	 *
> +	 * Emulate the FRED-defined redzone and stack alignment (128 bytes and
> +	 * 64 bytes respectively).
> +	 */
> +	sub $(FRED_CONFIG_REDZONE_AMOUNT << 6), %rsp
> +	and $FRED_STACK_FRAME_RSP_MASK, %rsp
> +
> +	/*
> +	* A FRED stack frame has extra 16 bytes of information pushed at the
> +	* regular stack top compared to an IDT stack frame.
> +	*/
> +	push $0         /* Reserved by FRED, must be 0 */
> +	push $0         /* FRED event data, 0 for NMI and external interrupts *=
/
> +	shl $32, %rax
> +	orq $__KERNEL_DS | $FRED_64_BIT_MODE, %ax
> +	push %rax	/* Vector (from the "caller") and DS */
> +
> +	push %rbp
> +	pushf
> +	push \cs_val

We need to push the RIP of the next instruction here. Or are you suggesting
we don't need to care about it because it may not be used to return from th=
e
callee?

As mentioned above, the return RIP is used when returning from NMI handling=
.

Or I totally missed a key idea to build a FRED stack frame?

Thanks!
  Xin

> +	push $0 /* FRED error code, 0 for NMI and external interrupts */
> +	PUSH_REGS
> +
> +	/* Load @pt_regs */
> +	movq    %rsp, %_ASM_ARG1
> +
> +	call \call_target
> +
> +	POP_REGS
> +
> +	/*
> +	 * "Restore" RSP from RBP, even though IRET has already unwound RSP
> to
> +	 * the correct value.  objtool doesn't know the callee will IRET and,
> +	 * without the explicit restore, thinks the stack is getting walloped.
> +	 * Using an unwind hint is problematic due to x86-64's dynamic
> alignment.
> +	 */
> +	mov %_ASM_BP, %_ASM_SP
> +	pop %_ASM_BP
> +	RET
> +.endm
> +#endif
> +