When runtime support for converting between 4-level and 5-level pagetables
was added to the kernel, the SME code that built pagetables was updated
to use the pagetable functions, e.g. p4d_offset(), etc., in order to
simplify the code. However, the use of the pagetable functions in early
boot code requires the use of the USE_EARLY_PGTABLE_L5 #define in order to
ensure that proper definition of pgtable_l5_enabled() is used.
Without the #define, pgtable_l5_enabled() is #defined as
cpu_feature_enabled(X86_FEATURE_LA57). In early boot, the CPU features
have not yet been discovered and populated, so pgtable_l5_enabled() will
return false even when 5-level paging is enabled. This causes the SME code
to always build 4-level pagetables to perform the in-place encryption.
If 5-level paging is enabled, switching to the SME pagetables results in
a page-fault that kills the boot.
Adding the #define results in pgtable_l5_enabled() using the
__pgtable_l5_enabled variable set in early boot and the SME code building
pagetables for the proper paging level.
Cc: <[email protected]> # 4.18.x
Fixes: aad983913d77 ("x86/mm/encrypt: Simplify sme_populate_pgd() and sme_populate_pgd_large()")
Signed-off-by: Tom Lendacky <[email protected]>
---
arch/x86/mm/mem_encrypt_identity.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c
index f8c612902038..3f0abb403340 100644
--- a/arch/x86/mm/mem_encrypt_identity.c
+++ b/arch/x86/mm/mem_encrypt_identity.c
@@ -27,6 +27,15 @@
#undef CONFIG_PARAVIRT_XXL
#undef CONFIG_PARAVIRT_SPINLOCKS
+/*
+ * This code runs before CPU feature bits are set. By default, the
+ * pgtable_l5_enabled() function uses bit X86_FEATURE_LA57 to determine if
+ * 5-level paging is active, so that won't work here. USE_EARLY_PGTABLE_L5
+ * is provided to handle this situation and, instead, use a variable that
+ * has been set by the early boot code.
+ */
+#define USE_EARLY_PGTABLE_L5
+
#include <linux/kernel.h>
#include <linux/mm.h>
#include <linux/mem_encrypt.h>
--
2.33.1
On Fri, Oct 15, 2021 at 12:24:16PM -0500, Tom Lendacky wrote:
> When runtime support for converting between 4-level and 5-level pagetables
> was added to the kernel, the SME code that built pagetables was updated
> to use the pagetable functions, e.g. p4d_offset(), etc., in order to
> simplify the code. However, the use of the pagetable functions in early
> boot code requires the use of the USE_EARLY_PGTABLE_L5 #define in order to
> ensure that proper definition of pgtable_l5_enabled() is used.
>
> Without the #define, pgtable_l5_enabled() is #defined as
> cpu_feature_enabled(X86_FEATURE_LA57). In early boot, the CPU features
> have not yet been discovered and populated, so pgtable_l5_enabled() will
> return false even when 5-level paging is enabled. This causes the SME code
> to always build 4-level pagetables to perform the in-place encryption.
> If 5-level paging is enabled, switching to the SME pagetables results in
> a page-fault that kills the boot.
>
> Adding the #define results in pgtable_l5_enabled() using the
> __pgtable_l5_enabled variable set in early boot and the SME code building
> pagetables for the proper paging level.
>
> Cc: <[email protected]> # 4.18.x
> Fixes: aad983913d77 ("x86/mm/encrypt: Simplify sme_populate_pgd() and sme_populate_pgd_large()")
> Signed-off-by: Tom Lendacky <[email protected]>
Acked-by: Kirill A. Shutemov <[email protected]>
--
Kirill A. Shutemov