Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <cover.1619458733.git.sathyanarayanan.kuppuswamy@linux.intel.com> <b6e4ea3dfe6648e111e0d9dfcce4174672f8b823.1619458733.git.sathyanarayanan.kuppuswamy@linux.intel.com>
In-Reply-To: <b6e4ea3dfe6648e111e0d9dfcce4174672f8b823.1619458733.git.sathyanarayanan.kuppuswamy@linux.intel.com>
From:   Dan Williams <dan.j.williams@intel.com>
Date:   Wed, 12 May 2021 20:23:17 -0700
Message-ID: <CAPcyv4jqr8vyh7BwYxW3QJ_ui_yH+iGniJYuUMEnTLWjiYsvPQ@mail.gmail.com>
Subject: Re: [RFC v2 23/32] x86/boot: Avoid unnecessary #VE during boot process
To:     Kuppuswamy Sathyanarayanan 
        <sathyanarayanan.kuppuswamy@linux.intel.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Andy Lutomirski <luto@kernel.org>,
        Dave Hansen <dave.hansen@intel.com>,
        Tony Luck <tony.luck@intel.com>,
        Andi Kleen <ak@linux.intel.com>,
        Kirill Shutemov <kirill.shutemov@linux.intel.com>,
        Kuppuswamy Sathyanarayanan <knsathya@kernel.org>,
        Raj Ashok <ashok.raj@intel.com>,
        Sean Christopherson <seanjc@google.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Mon, Apr 26, 2021 at 11:03 AM Kuppuswamy Sathyanarayanan
<sathyanarayanan.kuppuswamy@linux.intel.com> wrote:
>
> From: Sean Christopherson <sean.j.christopherson@intel.com>
>
> Skip writing EFER during secondary_startup_64() if the current value is
> also the desired value. This avoids a #VE when running as a TDX guest,
> as the TDX-Module does not allow writes to EFER (even when writing the
> current, fixed value).
>
> Also, preserve CR4.MCE instead of clearing it during boot to avoid a #VE
> when running as a TDX guest. The TDX-Module (effectively part of the
> hypervisor) requires CR4.MCE to be set at all times and injects a #VE
> if the guest attempts to clear CR4.MCE.
>
> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> Reviewed-by: Andi Kleen <ak@linux.intel.com>
> Signed-off-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
> ---
>  arch/x86/boot/compressed/head_64.S |  5 ++++-
>  arch/x86/kernel/head_64.S          | 13 +++++++++++--
>  2 files changed, 15 insertions(+), 3 deletions(-)
>
> diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S
> index 37c2f37d4a0d..2d79e5f97360 100644
> --- a/arch/x86/boot/compressed/head_64.S
> +++ b/arch/x86/boot/compressed/head_64.S
> @@ -622,7 +622,10 @@ SYM_CODE_START(trampoline_32bit_src)
>         popl    %ecx
>
>         /* Enable PAE and LA57 (if required) paging modes */
> -       movl    $X86_CR4_PAE, %eax
> +       movl    %cr4, %eax
> +       /* Clearing CR4.MCE will #VE on TDX guests.  Leave it alone. */
> +       andl    $X86_CR4_MCE, %eax
> +       orl     $X86_CR4_PAE, %eax
>         testl   %edx, %edx
>         jz      1f
>         orl     $X86_CR4_LA57, %eax
> diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
> index 04bddaaba8e2..92c77cf75542 100644
> --- a/arch/x86/kernel/head_64.S
> +++ b/arch/x86/kernel/head_64.S
> @@ -141,7 +141,10 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL)
>  1:
>
>         /* Enable PAE mode, PGE and LA57 */
> -       movl    $(X86_CR4_PAE | X86_CR4_PGE), %ecx
> +       movq    %cr4, %rcx
> +       /* Clearing CR4.MCE will #VE on TDX guests.  Leave it alone. */
> +       andl    $X86_CR4_MCE, %ecx
> +       orl     $(X86_CR4_PAE | X86_CR4_PGE), %ecx
>  #ifdef CONFIG_X86_5LEVEL
>         testl   $1, __pgtable_l5_enabled(%rip)
>         jz      1f
> @@ -229,13 +232,19 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL)
>         /* Setup EFER (Extended Feature Enable Register) */
>         movl    $MSR_EFER, %ecx
>         rdmsr
> +       movl    %eax, %edx

Maybe comment that EFER is being saved here to check if the following
enables are nops, but not a big deal.

Reviewed-by: Dan Williams <dan.j.williams@intel.com>

...modulo whether the EFER wrmsr avoidance in PATCH 21 should move here.

>         btsl    $_EFER_SCE, %eax        /* Enable System Call */
>         btl     $20,%edi                /* No Execute supported? */
>         jnc     1f
>         btsl    $_EFER_NX, %eax
>         btsq    $_PAGE_BIT_NX,early_pmd_flags(%rip)
> -1:     wrmsr                           /* Make changes effective */
>
> +       /* Skip the WRMSR if the current value matches the desired value. */
> +1:     cmpl    %edx, %eax
> +       je      1f
> +       xor     %edx, %edx
> +       wrmsr                           /* Make changes effective */
> +1:
>         /* Setup cr0 */
>         movl    $CR0_STATE, %eax
>         /* Make changes effective */
> --
> 2.25.1
>