Hi all,
As posting this version, I wanted to make sure these code changes were
acknowledgeable at first:
The previous enabling process has been paused to address vulnerabilities
[1][2] that could compromise Key Locker's ability to protect AES keys.
Now, with the mainlining of mitigations [3][4], patches (Patch 10-11)
were added to ensure the application of these mitigations.
During this period, there was a significant change in the mainline commit
b81fac906a8f ("x86/fpu: Move FPU initialization into
arch_cpu_finalize_init()"). This affected Key Locker's initialization
code, which clobbers XMM registers for loading a wrapping key, as it
depends on FPU initialization.
In this revision, the setup code was adjusted to separate the
initialization part to be invoked during arch_initcall(). The remaining
code for copying the wrapping key from the backup resides in the
identify_cpu() -> setup_keylocker() path. This separation simplifies the
code and resolves an issue with hotplug.
The remaining changes mainly focus on the AES crypto driver, addressing
feedback from Eric. Notably, while doing so, it was realized better to
disallow a module build. Key Locker's AES instructions do not support
192-bit keys. Supporting a module build would require exporting some
AES-NI functions, leading to performance-impacting indirect calls. I
think we can revisit module support later if necessary.
Then, the following is a summary of changes per patch since v8 [6]:
PATCH7-8:
* Invoke the setup code via arch_initcall() due to upstream changes
delaying the FPU setup.
PATCH9-11:
* Add new patches for security and hotplug support clarification
PATCH12:
* Drop the "nokeylocker" option. (Borislav Petkov)
PATCH13:
* Introduce 'union x86_aes_ctx'. (Eric Biggers)
* Ensure 'inline' for wrapper functions.
PATCH14:
* Combine the XTS enc/dec assembly code in a macro. (Eric Biggers)
* Define setkey() as void instead of returning 'int'. (Eric Biggers)
* Rearrange the assembly code to reduce jumps, especially for success
cases. (Eric Biggers)
* Update the changelog for clarification. (Eric Biggers)
* Exclude module build.
This series is based on my AES-NI setkey() cleanup [7], which has been
recently merged into the crypto repository [8], and I thought it was
better to go first. You can also find this series here:
git://github.com/intel-staging/keylocker.git kl-v9
Thanks,
Chang
[1] Gather Data Sampling (GDS)
https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/gather-data-sampling.html
[2] Register File Data Sampling (RFDS)
https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/advisory-guidance/register-file-data-sampling.html
[3] Mainlining of GDS mitigation
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=64094e7e3118aff4b0be8ff713c242303e139834
[4] Mainlining of RFDS Mitigation
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0e33cf955f07e3991e45109cb3e29fbc9ca51d06
[5] Initialize FPU late
https://lore.kernel.org/lkml/[email protected]/
[6] V8: https://lore.kernel.org/lkml/[email protected]/
[7] https://lore.kernel.org/lkml/[email protected]/
[8] git://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git
Chang S. Bae (14):
Documentation/x86: Document Key Locker
x86/cpufeature: Enumerate Key Locker feature
x86/insn: Add Key Locker instructions to the opcode map
x86/asm: Add a wrapper function for the LOADIWKEY instruction
x86/msr-index: Add MSRs for Key Locker wrapping key
x86/keylocker: Define Key Locker CPUID leaf
x86/cpu/keylocker: Load a wrapping key at boot time
x86/PM/keylocker: Restore the wrapping key on the resume from ACPI
S3/4
x86/hotplug/keylocker: Ensure wrapping key backup capability
x86/cpu/keylocker: Check Gather Data Sampling mitigation
x86/cpu/keylocker: Check Register File Data Sampling mitigation
x86/Kconfig: Add a configuration for Key Locker
crypto: x86/aes - Prepare for new AES-XTS implementation
crypto: x86/aes-kl - Implement the AES-XTS algorithm
Documentation/arch/x86/index.rst | 1 +
Documentation/arch/x86/keylocker.rst | 96 +++++
arch/x86/Kconfig | 3 +
arch/x86/Kconfig.assembler | 5 +
arch/x86/crypto/Kconfig | 17 +
arch/x86/crypto/Makefile | 3 +
arch/x86/crypto/aes-helper_asm.S | 22 ++
arch/x86/crypto/aes-helper_glue.h | 168 ++++++++
arch/x86/crypto/aeskl-intel_asm.S | 412 ++++++++++++++++++++
arch/x86/crypto/aeskl-intel_glue.c | 187 +++++++++
arch/x86/crypto/aeskl-intel_glue.h | 35 ++
arch/x86/crypto/aesni-intel_asm.S | 47 +--
arch/x86/crypto/aesni-intel_glue.c | 193 ++-------
arch/x86/crypto/aesni-intel_glue.h | 40 ++
arch/x86/include/asm/cpufeatures.h | 1 +
arch/x86/include/asm/disabled-features.h | 8 +-
arch/x86/include/asm/keylocker.h | 42 ++
arch/x86/include/asm/msr-index.h | 6 +
arch/x86/include/asm/special_insns.h | 28 ++
arch/x86/include/uapi/asm/processor-flags.h | 2 +
arch/x86/kernel/Makefile | 1 +
arch/x86/kernel/cpu/common.c | 4 +-
arch/x86/kernel/cpu/cpuid-deps.c | 1 +
arch/x86/kernel/keylocker.c | 219 +++++++++++
arch/x86/lib/x86-opcode-map.txt | 11 +-
arch/x86/power/cpu.c | 2 +
tools/arch/x86/lib/x86-opcode-map.txt | 11 +-
27 files changed, 1363 insertions(+), 202 deletions(-)
create mode 100644 Documentation/arch/x86/keylocker.rst
create mode 100644 arch/x86/crypto/aes-helper_asm.S
create mode 100644 arch/x86/crypto/aes-helper_glue.h
create mode 100644 arch/x86/crypto/aeskl-intel_asm.S
create mode 100644 arch/x86/crypto/aeskl-intel_glue.c
create mode 100644 arch/x86/crypto/aeskl-intel_glue.h
create mode 100644 arch/x86/crypto/aesni-intel_glue.h
create mode 100644 arch/x86/include/asm/keylocker.h
create mode 100644 arch/x86/kernel/keylocker.c
base-commit: 3a447c31d337bdec7fbc605a7a1e00aff4c492d0
--
2.34.1
Key Locker is a CPU feature to minimize exposure of clear-text key
material. An encoded form, called 'key handle', is referenced for data
encryption or decryption instead of accessing the clear text key.
A wrapping key loaded in the CPU's software-inaccessible state is used
to transform a user key into a key handle. On rarely unexpected
hardware failure, the key could be lost.
Here enumerate this hardware capability. It will not be shown up in
/proc/cpuinfo as userspace usage is not supported. This is because
there is no ABI to coordinate the wrapping-key failure.
The feature supports Advanced Encryption Standard (AES) cipher
algorithm with new SIMD instruction set like its predecessor (AES-NI).
Mark the feature having dependency on XMM2 like AES-NI has. The new AES
implementation will be in the crypto library.
Add X86_FEATURE_KEYLOCKER to the disabled feature list. It will be
enabled by a new Kconfig option.
Signed-off-by: Chang S. Bae <[email protected]>
Reviewed-by: Dan Williams <[email protected]>
---
Changes from v6:
* Massage the changelog -- re-organize the change descriptions
Changes from RFC v2:
* Do not publish the feature flag to userspace.
* Update the changelog.
Changes from RFC v1:
* Updated the changelog.
---
arch/x86/include/asm/cpufeatures.h | 1 +
arch/x86/include/asm/disabled-features.h | 8 +++++++-
arch/x86/include/uapi/asm/processor-flags.h | 2 ++
arch/x86/kernel/cpu/cpuid-deps.c | 1 +
4 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index f0337f7bcf16..dd30435af487 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -399,6 +399,7 @@
#define X86_FEATURE_AVX512_VPOPCNTDQ (16*32+14) /* POPCNT for vectors of DW/QW */
#define X86_FEATURE_LA57 (16*32+16) /* 5-level page tables */
#define X86_FEATURE_RDPID (16*32+22) /* RDPID instruction */
+#define X86_FEATURE_KEYLOCKER (16*32+23) /* "" Key Locker */
#define X86_FEATURE_BUS_LOCK_DETECT (16*32+24) /* Bus Lock detect */
#define X86_FEATURE_CLDEMOTE (16*32+25) /* CLDEMOTE instruction */
#define X86_FEATURE_MOVDIRI (16*32+27) /* MOVDIRI instruction */
diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h
index da4054fbf533..14aa6dc3b846 100644
--- a/arch/x86/include/asm/disabled-features.h
+++ b/arch/x86/include/asm/disabled-features.h
@@ -38,6 +38,12 @@
# define DISABLE_OSPKE (1<<(X86_FEATURE_OSPKE & 31))
#endif /* CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS */
+#ifdef CONFIG_X86_KEYLOCKER
+# define DISABLE_KEYLOCKER 0
+#else
+# define DISABLE_KEYLOCKER (1<<(X86_FEATURE_KEYLOCKER & 31))
+#endif /* CONFIG_X86_KEYLOCKER */
+
#ifdef CONFIG_X86_5LEVEL
# define DISABLE_LA57 0
#else
@@ -150,7 +156,7 @@
#define DISABLED_MASK14 0
#define DISABLED_MASK15 0
#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \
- DISABLE_ENQCMD)
+ DISABLE_ENQCMD|DISABLE_KEYLOCKER)
#define DISABLED_MASK17 0
#define DISABLED_MASK18 (DISABLE_IBT)
#define DISABLED_MASK19 (DISABLE_SEV_SNP)
diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h
index f1a4adc78272..a24f7cb2cd68 100644
--- a/arch/x86/include/uapi/asm/processor-flags.h
+++ b/arch/x86/include/uapi/asm/processor-flags.h
@@ -128,6 +128,8 @@
#define X86_CR4_PCIDE _BITUL(X86_CR4_PCIDE_BIT)
#define X86_CR4_OSXSAVE_BIT 18 /* enable xsave and xrestore */
#define X86_CR4_OSXSAVE _BITUL(X86_CR4_OSXSAVE_BIT)
+#define X86_CR4_KEYLOCKER_BIT 19 /* enable Key Locker */
+#define X86_CR4_KEYLOCKER _BITUL(X86_CR4_KEYLOCKER_BIT)
#define X86_CR4_SMEP_BIT 20 /* enable SMEP support */
#define X86_CR4_SMEP _BITUL(X86_CR4_SMEP_BIT)
#define X86_CR4_SMAP_BIT 21 /* enable SMAP support */
diff --git a/arch/x86/kernel/cpu/cpuid-deps.c b/arch/x86/kernel/cpu/cpuid-deps.c
index b7174209d855..820dcf35eca9 100644
--- a/arch/x86/kernel/cpu/cpuid-deps.c
+++ b/arch/x86/kernel/cpu/cpuid-deps.c
@@ -84,6 +84,7 @@ static const struct cpuid_dep cpuid_deps[] = {
{ X86_FEATURE_SHSTK, X86_FEATURE_XSAVES },
{ X86_FEATURE_FRED, X86_FEATURE_LKGS },
{ X86_FEATURE_FRED, X86_FEATURE_WRMSRNS },
+ { X86_FEATURE_KEYLOCKER, X86_FEATURE_XMM2 },
{}
};
--
2.34.1
The x86 instruction decoder needs to know these new instructions that
are going to be used in the crypto library as well as the x86 core
code. Add the following:
LOADIWKEY:
Load a CPU-internal wrapping key.
ENCODEKEY128:
Wrap a 128-bit AES key to a key handle.
ENCODEKEY256:
Wrap a 256-bit AES key to a key handle.
AESENC128KL:
Encrypt a 128-bit block of data using a 128-bit AES key
indicated by a key handle.
AESENC256KL:
Encrypt a 128-bit block of data using a 256-bit AES key
indicated by a key handle.
AESDEC128KL:
Decrypt a 128-bit block of data using a 128-bit AES key
indicated by a key handle.
AESDEC256KL:
Decrypt a 128-bit block of data using a 256-bit AES key
indicated by a key handle.
AESENCWIDE128KL:
Encrypt 8 128-bit blocks of data using a 128-bit AES key
indicated by a key handle.
AESENCWIDE256KL:
Encrypt 8 128-bit blocks of data using a 256-bit AES key
indicated by a key handle.
AESDECWIDE128KL:
Decrypt 8 128-bit blocks of data using a 128-bit AES key
indicated by a key handle.
AESDECWIDE256KL:
Decrypt 8 128-bit blocks of data using a 256-bit AES key
indicated by a key handle.
The detail can be found in Intel Software Developer Manual.
Signed-off-by: Chang S. Bae <[email protected]>
Reviewed-by: Dan Williams <[email protected]>
---
Changes from v6:
* Massage the changelog -- add the reason a bit.
Changes from RFC v1:
* Separated out the LOADIWKEY addition in a new patch.
* Included AES instructions to avoid warning messages when the AES Key
Locker module is built.
---
arch/x86/lib/x86-opcode-map.txt | 11 +++++++----
tools/arch/x86/lib/x86-opcode-map.txt | 11 +++++++----
2 files changed, 14 insertions(+), 8 deletions(-)
diff --git a/arch/x86/lib/x86-opcode-map.txt b/arch/x86/lib/x86-opcode-map.txt
index 12af572201a2..c94988d5130d 100644
--- a/arch/x86/lib/x86-opcode-map.txt
+++ b/arch/x86/lib/x86-opcode-map.txt
@@ -800,11 +800,12 @@ cb: sha256rnds2 Vdq,Wdq | vrcp28ss/d Vx,Hx,Wx (66),(ev)
cc: sha256msg1 Vdq,Wdq | vrsqrt28ps/d Vx,Wx (66),(ev)
cd: sha256msg2 Vdq,Wdq | vrsqrt28ss/d Vx,Hx,Wx (66),(ev)
cf: vgf2p8mulb Vx,Wx (66)
+d8: AESENCWIDE128KL Qpi (F3),(000),(00B) | AESENCWIDE256KL Qpi (F3),(000),(10B) | AESDECWIDE128KL Qpi (F3),(000),(01B) | AESDECWIDE256KL Qpi (F3),(000),(11B)
db: VAESIMC Vdq,Wdq (66),(v1)
-dc: vaesenc Vx,Hx,Wx (66)
-dd: vaesenclast Vx,Hx,Wx (66)
-de: vaesdec Vx,Hx,Wx (66)
-df: vaesdeclast Vx,Hx,Wx (66)
+dc: vaesenc Vx,Hx,Wx (66) | LOADIWKEY Vx,Hx (F3) | AESENC128KL Vpd,Qpi (F3)
+dd: vaesenclast Vx,Hx,Wx (66) | AESDEC128KL Vpd,Qpi (F3)
+de: vaesdec Vx,Hx,Wx (66) | AESENC256KL Vpd,Qpi (F3)
+df: vaesdeclast Vx,Hx,Wx (66) | AESDEC256KL Vpd,Qpi (F3)
f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2)
f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2)
f2: ANDN Gy,By,Ey (v)
@@ -814,6 +815,8 @@ f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSSD/Q My,
f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v)
f8: MOVDIR64B Gv,Mdqq (66) | ENQCMD Gv,Mdqq (F2) | ENQCMDS Gv,Mdqq (F3)
f9: MOVDIRI My,Gy
+fa: ENCODEKEY128 Ew,Ew (F3)
+fb: ENCODEKEY256 Ew,Ew (F3)
EndTable
Table: 3-byte opcode 2 (0x0f 0x3a)
diff --git a/tools/arch/x86/lib/x86-opcode-map.txt b/tools/arch/x86/lib/x86-opcode-map.txt
index 12af572201a2..c94988d5130d 100644
--- a/tools/arch/x86/lib/x86-opcode-map.txt
+++ b/tools/arch/x86/lib/x86-opcode-map.txt
@@ -800,11 +800,12 @@ cb: sha256rnds2 Vdq,Wdq | vrcp28ss/d Vx,Hx,Wx (66),(ev)
cc: sha256msg1 Vdq,Wdq | vrsqrt28ps/d Vx,Wx (66),(ev)
cd: sha256msg2 Vdq,Wdq | vrsqrt28ss/d Vx,Hx,Wx (66),(ev)
cf: vgf2p8mulb Vx,Wx (66)
+d8: AESENCWIDE128KL Qpi (F3),(000),(00B) | AESENCWIDE256KL Qpi (F3),(000),(10B) | AESDECWIDE128KL Qpi (F3),(000),(01B) | AESDECWIDE256KL Qpi (F3),(000),(11B)
db: VAESIMC Vdq,Wdq (66),(v1)
-dc: vaesenc Vx,Hx,Wx (66)
-dd: vaesenclast Vx,Hx,Wx (66)
-de: vaesdec Vx,Hx,Wx (66)
-df: vaesdeclast Vx,Hx,Wx (66)
+dc: vaesenc Vx,Hx,Wx (66) | LOADIWKEY Vx,Hx (F3) | AESENC128KL Vpd,Qpi (F3)
+dd: vaesenclast Vx,Hx,Wx (66) | AESDEC128KL Vpd,Qpi (F3)
+de: vaesdec Vx,Hx,Wx (66) | AESENC256KL Vpd,Qpi (F3)
+df: vaesdeclast Vx,Hx,Wx (66) | AESDEC256KL Vpd,Qpi (F3)
f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2)
f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2)
f2: ANDN Gy,By,Ey (v)
@@ -814,6 +815,8 @@ f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSSD/Q My,
f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v)
f8: MOVDIR64B Gv,Mdqq (66) | ENQCMD Gv,Mdqq (F2) | ENQCMDS Gv,Mdqq (F3)
f9: MOVDIRI My,Gy
+fa: ENCODEKEY128 Ew,Ew (F3)
+fb: ENCODEKEY256 Ew,Ew (F3)
EndTable
Table: 3-byte opcode 2 (0x0f 0x3a)
--
2.34.1
Key Locker introduces a CPU-internal wrapping key to encode a user key
to a key handle. Then a key handle is referenced instead of the plain
text key.
LOADIWKEY loads a wrapping key in the software-inaccessible CPU state.
It operates only in kernel mode.
The kernel will use this to load a new key at boot time. Establish an
accessor for the feature setup, and define struct iwkey to pass a key
value.
Signed-off-by: Chang S. Bae <[email protected]>
Reviewed-by: Dan Williams <[email protected]>
---
Changes from v6:
* Massage the changelog -- clarify the reason and the changes a bit.
Changes from v5:
* Fix a typo: kernel_cpu_begin() -> kernel_fpu_begin()
Changes from RFC v2:
* Separate out the code as a new patch.
* Improve the usability with the new struct as an argument. (Dan
Williams)
Previously, Dan questioned the necessity of 'WARN_ON(!irq_fpu_usable())'
in the load_xmm_iwkey() function. However, it's worth noting that the
function comment emphasizes the caller's responsibility for invoking
kernel_fpu_begin(), which effectively performs the sanity check through
kernel_fpu_begin_mask().
---
arch/x86/include/asm/keylocker.h | 25 +++++++++++++++++++++++++
arch/x86/include/asm/special_insns.h | 28 ++++++++++++++++++++++++++++
2 files changed, 53 insertions(+)
create mode 100644 arch/x86/include/asm/keylocker.h
diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h
new file mode 100644
index 000000000000..4e731f577c50
--- /dev/null
+++ b/arch/x86/include/asm/keylocker.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#ifndef _ASM_KEYLOCKER_H
+#define _ASM_KEYLOCKER_H
+
+#ifndef __ASSEMBLY__
+
+#include <asm/fpu/types.h>
+
+/**
+ * struct iwkey - A temporary wrapping key storage.
+ * @integrity_key: A 128-bit key used to verify the integrity of
+ * key handles
+ * @encryption_key: A 256-bit encryption key used for wrapping and
+ * unwrapping clear text keys.
+ *
+ * This storage should be flushed immediately after being loaded.
+ */
+struct iwkey {
+ struct reg_128_bit integrity_key;
+ struct reg_128_bit encryption_key[2];
+};
+
+#endif /*__ASSEMBLY__ */
+#endif /* _ASM_KEYLOCKER_H */
diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h
index 2e9fc5c400cd..65267013f1e1 100644
--- a/arch/x86/include/asm/special_insns.h
+++ b/arch/x86/include/asm/special_insns.h
@@ -9,6 +9,7 @@
#include <linux/errno.h>
#include <linux/irqflags.h>
#include <linux/jump_label.h>
+#include <asm/keylocker.h>
/*
* The compiler should not reorder volatile asm statements with respect to each
@@ -301,6 +302,33 @@ static __always_inline void tile_release(void)
asm volatile(".byte 0xc4, 0xe2, 0x78, 0x49, 0xc0");
}
+/**
+ * load_xmm_iwkey - Load a CPU-internal wrapping key into XMM registers.
+ * @key: A pointer to a struct iwkey containing the key data.
+ *
+ * The caller is responsible for invoking kernel_fpu_begin() before.
+ */
+static inline void load_xmm_iwkey(struct iwkey *key)
+{
+ struct reg_128_bit zeros = { 0 };
+
+ asm volatile ("movdqu %0, %%xmm0; movdqu %1, %%xmm1; movdqu %2, %%xmm2;"
+ :: "m"(key->integrity_key), "m"(key->encryption_key[0]),
+ "m"(key->encryption_key[1]));
+
+ /*
+ * 'LOADIWKEY %xmm1,%xmm2' loads a key from XMM0-2 into a
+ * software-invisible CPU state. With zero in EAX, CPU does not
+ * perform hardware randomization and allows key backup.
+ *
+ * This instruction is supported by binutils >= 2.36.
+ */
+ asm volatile (".byte 0xf3,0x0f,0x38,0xdc,0xd1" :: "a"(0));
+
+ asm volatile ("movdqu %0, %%xmm0; movdqu %0, %%xmm1; movdqu %0, %%xmm2;"
+ :: "m"(zeros));
+}
+
#endif /* __KERNEL__ */
#endif /* _ASM_X86_SPECIAL_INSNS_H */
--
2.34.1
The primary use case for the feature is bare metal dm-crypt. The key
needs to be restored properly on wakeup, as dm-crypt does not prompt
for the key on resume from suspend. Even if the prompt performs for
unlocking the volume, where the hibernation image is stored, it still
expects to reuse the key handles within the hibernation image once it
is loaded.
== Wrapping-key Restore ==
To meet dm-crypt's expectations, the key handles in the suspend-image has
to remain valid after resuming from an S-state. However, when the system
enters ACPI S3 or S4 sleep states, the wrapping key is discarded.
Key Locker provides a mechanism to back up the wrapping key in
non-volatile storage. Therefore, upon boot, request a backup of the
wrapping key and copy it back to each CPU upon wakeup. If the backup
mechanism is unavailable, disable the feature unless CONFIG_SUSPEND=n.
== Restore Failure ==
In the event of a key restore failure, the kernel proceeds with an
initialized wrapping key state. This action invalidates any key handles
present in the suspend-image, leading to I/O errors in dm-crypt
operations.
However, data integrity remains intact, and access is restored with new
handles created by the new wrapping key at the next boot. At least,
manage a feature-specific flag to communicate with the crypto
implementation, ensuring to stop using AES instructions upon the key
restore failure, instead of abruptly disabling the feature.
== Off-states ==
While the backup may persist in non-volatile media across S5 and G3 "off"
states, it is neither architecturally guaranteed nor expected by
dm-crypt. Therefore, a reboot can address this scenario with a new
wrapping key, as dm-crypt prompts for the key whenever the volume is
started.
Signed-off-by: Chang S. Bae <[email protected]>
Acked-by: Rafael J. Wysocki <[email protected]>
Cc: Eric Biggers <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: Sangwhan Moon <[email protected]>
Cc: Dan Williams <[email protected]>
---
Changes from v8:
* Rebase on the previous patch (patch7) changes, separating the wrapping
key restoration code from the initial load. Previously, the
identify_cpu() -> setup_keylocker() sequence in the hotplug path could
hit __init code, leading to an explosion. This change removes the
initialization code from the hotplug path. (Sangwhan Moon)
* Turn copy_keylocker() to return bool for simplification.
* Rename the flag for clarity: 'valid_kl' -> 'valid_wrapping_key'.
* Don't export symbol for valid_keylocker(), as AES-KL will be built-in.
(see patch14 for detail).
* Tweak code comments and the changelog.
* Revoke the review tag as the code change is significant.
Changes from v6:
* Limit the symbol export only when needed.
* Improve the coding style -- reduce an indent after
'if() { ... return; }'. (Eric Biggers)
* Fix the coding style -- reduce an indent after if() {...return;}.
(Eric Biggers) Tweak the comment along with that.
* Improve the function prototype, instead of using a macro. (Eric
Biggers and Dave Hansen)
* Update the documentation:
- Massage the changelog to clarify the problem-and-solution by
sections
- Clarify the comment about the key restore failure.
Changes from v5:
* Fix the 'valid_kl' flag not to be set when the feature is disabled.
(Reported by Marvin Hsu [email protected]) Add the function
comment about this.
* Improve the error handling in setup_keylocker(). All the error cases
fall through the end that disables the feature. Otherwise, all the
successful cases return immediately.
Changes from v4:
* Update the changelog and title. (Rafael Wysocki)
Changes from v3:
* Fix the build issue with !X86_KEYLOCKER. (Eric Biggers)
Changes from RFC v2:
* Change the backup key failure handling. (Dan Williams)
Changes from RFC v1:
* Folded the warning message into the if condition check. (Rafael
Wysocki)
* Rebase on the changes of the previous patches.
* Added error code for key restoration failures.
* Moved the restore helper.
* Added function descriptions.
---
arch/x86/include/asm/keylocker.h | 10 ++++
arch/x86/kernel/cpu/common.c | 4 +-
arch/x86/kernel/keylocker.c | 88 ++++++++++++++++++++++++++++++++
arch/x86/power/cpu.c | 2 +
4 files changed, 103 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h
index 1213d273c369..c93102101c41 100644
--- a/arch/x86/include/asm/keylocker.h
+++ b/arch/x86/include/asm/keylocker.h
@@ -28,5 +28,15 @@ struct iwkey {
#define KEYLOCKER_CPUID_EBX_WIDE BIT(2)
#define KEYLOCKER_CPUID_EBX_BACKUP BIT(4)
+#ifdef CONFIG_X86_KEYLOCKER
+void setup_keylocker(void);
+void restore_keylocker(void);
+extern bool valid_keylocker(void);
+#else
+static inline void setup_keylocker(void) { }
+static inline void restore_keylocker(void) { }
+static inline bool valid_keylocker(void) { return false; }
+#endif
+
#endif /*__ASSEMBLY__ */
#endif /* _ASM_KEYLOCKER_H */
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 5c1e6d6be267..bfbb1ca64664 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -62,6 +62,7 @@
#include <asm/intel-family.h>
#include <asm/cpu_device_id.h>
#include <asm/fred.h>
+#include <asm/keylocker.h>
#include <asm/uv/uv.h>
#include <asm/ia32.h>
#include <asm/set_memory.h>
@@ -1826,10 +1827,11 @@ static void identify_cpu(struct cpuinfo_x86 *c)
/* Disable the PN if appropriate */
squash_the_stupid_serial_number(c);
- /* Set up SMEP/SMAP/UMIP */
+ /* Setup various Intel-specific CPU security features */
setup_smep(c);
setup_smap(c);
setup_umip(c);
+ setup_keylocker();
/* Enable FSGSBASE instructions if available. */
if (cpu_has(c, X86_FEATURE_FSGSBASE)) {
diff --git a/arch/x86/kernel/keylocker.c b/arch/x86/kernel/keylocker.c
index 0d6b715baf1e..d5d11d0263b7 100644
--- a/arch/x86/kernel/keylocker.c
+++ b/arch/x86/kernel/keylocker.c
@@ -9,10 +9,24 @@
#include <asm/fpu/api.h>
#include <asm/keylocker.h>
+#include <asm/msr.h>
#include <asm/processor.h>
static struct iwkey wrapping_key __initdata;
+/*
+ * This flag is set when a wrapping key is successfully loaded. If a key
+ * restoration fails, it is reset. This state is exported to the crypto
+ * library, indicating whether Key Locker is usable. Thus, the feature
+ * can be soft-disabled based on this flag.
+ */
+static bool valid_wrapping_key;
+
+bool valid_keylocker(void)
+{
+ return valid_wrapping_key;
+}
+
static void __init generate_keylocker_data(void)
{
get_random_bytes(&wrapping_key.integrity_key, sizeof(wrapping_key.integrity_key));
@@ -37,9 +51,69 @@ static void __init load_keylocker(struct work_struct *unused)
kernel_fpu_end();
}
+/**
+ * copy_keylocker - Copy the wrapping key from the backup.
+ *
+ * Returns: true if successful, otherwise false.
+ */
+static bool copy_keylocker(void)
+{
+ u64 status;
+
+ wrmsrl(MSR_IA32_COPY_IWKEY_TO_LOCAL, 1);
+ rdmsrl(MSR_IA32_IWKEY_COPY_STATUS, status);
+ return !!(status & BIT(0));
+}
+
+/*
+ * On wakeup, APs copy a wrapping key after the boot CPU verifies a valid
+ * backup status through restore_keylocker(). Subsequently, they adhere
+ * to the error handling protocol by invalidating the flag.
+ */
+void setup_keylocker(void)
+{
+ if (!valid_wrapping_key)
+ return;
+
+ cr4_set_bits(X86_CR4_KEYLOCKER);
+
+ if (copy_keylocker())
+ return;
+
+ pr_err_once("x86/keylocker: Invalid copy status.\n");
+ valid_wrapping_key = false;
+}
+
+/* The boot CPU restores the wrapping key in the first place on wakeup. */
+void restore_keylocker(void)
+{
+ u64 backup_status;
+
+ if (!valid_wrapping_key)
+ return;
+
+ rdmsrl(MSR_IA32_IWKEY_BACKUP_STATUS, backup_status);
+ if (backup_status & BIT(0)) {
+ if (copy_keylocker())
+ return;
+ pr_err("x86/keylocker: Invalid copy state.\n");
+ } else {
+ pr_err("x86/keylocker: The key backup access failed with %s.\n",
+ (backup_status & BIT(2)) ? "read error" : "invalid status");
+ }
+
+ /*
+ * Invalidate the feature via this flag to indicate that the
+ * crypto code should voluntarily stop using the feature, rather
+ * than abruptly disabling it.
+ */
+ valid_wrapping_key = false;
+}
+
static int __init init_keylocker(void)
{
u32 eax, ebx, ecx, edx;
+ bool backup_available;
if (!cpu_feature_enabled(X86_FEATURE_KEYLOCKER))
goto disable;
@@ -59,9 +133,23 @@ static int __init init_keylocker(void)
goto clear_cap;
}
+ /*
+ * The backup is critical for restoring the wrapping key upon
+ * wakeup.
+ */
+ backup_available = !!(ebx & KEYLOCKER_CPUID_EBX_BACKUP);
+ if (!backup_available && IS_ENABLED(CONFIG_SUSPEND)) {
+ pr_debug("x86/keylocker: No key backup with possible S3/4.\n");
+ goto clear_cap;
+ }
+
generate_keylocker_data();
schedule_on_each_cpu(load_keylocker);
destroy_keylocker_data();
+ valid_wrapping_key = true;
+
+ if (backup_available)
+ wrmsrl(MSR_IA32_BACKUP_IWKEY_TO_PLATFORM, 1);
pr_info_once("x86/keylocker: Enabled.\n");
return 0;
diff --git a/arch/x86/power/cpu.c b/arch/x86/power/cpu.c
index 63230ff8cf4f..e99be45354cd 100644
--- a/arch/x86/power/cpu.c
+++ b/arch/x86/power/cpu.c
@@ -27,6 +27,7 @@
#include <asm/mmu_context.h>
#include <asm/cpu_device_id.h>
#include <asm/microcode.h>
+#include <asm/keylocker.h>
#ifdef CONFIG_X86_32
__visible unsigned long saved_context_ebx;
@@ -264,6 +265,7 @@ static void notrace __restore_processor_state(struct saved_context *ctxt)
x86_platform.restore_sched_clock_state();
cache_bp_restore();
perf_restore_debug_store();
+ restore_keylocker();
c = &cpu_data(smp_processor_id());
if (cpu_has(c, X86_FEATURE_MSR_IA32_FEAT_CTL))
--
2.34.1
Gather Data Sampling is a transient execution side channel issue in some
CPU models. The stale data in registers is not guaranteed as secure when
this vulnerability is not addressed.
In the Key Locker usage during AES transformations, the temporary storage
of the original key in registers poses a risk. The key material can be
staled in some implementations, leading to susceptibility to leakage of
the AES key.
To mitigate this vulnerability, a qualified microcode image must be
applied. Software then verifies the mitigation state using MSRs. Add code
to ensure that the mitigation is installed and securely locked. Disable
the feature, otherwise.
Signed-off-by: Chang S. Bae <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: Pawan Gupta <[email protected]>
---
Changes from v8:
* Add as a new patch.
Note that the code follows the guidance from [1]:
"Intel recommends that system software does not enable Key Locker (by
setting CR4.KL) unless the GDS mitigation is enabled
(IA32_MCU_OPT_CTRL[GDS_MITG_DIS] (bit 4) is 0) and locked (IA32_MCU_OPT_CTRL
[GDS_MITG_LOCK](bit 5) is 1)."
[1] https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/gather-data-sampling.html
---
arch/x86/kernel/keylocker.c | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/arch/x86/kernel/keylocker.c b/arch/x86/kernel/keylocker.c
index 1b57e11d93ad..d4f3aa65ea8a 100644
--- a/arch/x86/kernel/keylocker.c
+++ b/arch/x86/kernel/keylocker.c
@@ -7,6 +7,7 @@
#include <linux/random.h>
#include <linux/string.h>
+#include <asm/cpu.h>
#include <asm/fpu/api.h>
#include <asm/keylocker.h>
#include <asm/msr.h>
@@ -112,6 +113,37 @@ void restore_keylocker(void)
valid_wrapping_key = false;
}
+/*
+ * The mitigation is implemented at a microcode level. Ensure that the
+ * microcode update is applied and the mitigation is locked.
+ */
+static bool __init have_gds_mitigation(void)
+{
+ u64 mcu_ctrl;
+
+ /* GDS_CTRL is set if new microcode is loaded. */
+ if (!(x86_read_arch_cap_msr() & ARCH_CAP_GDS_CTRL))
+ goto vulnerable;
+
+ /* If GDS_MITG_LOCKED is set, GDS_MITG_DIS is forced to 0. */
+ rdmsrl(MSR_IA32_MCU_OPT_CTRL, mcu_ctrl);
+ if (mcu_ctrl & GDS_MITG_LOCKED)
+ return true;
+
+vulnerable:
+ pr_warn("x86/keylocker: Susceptible to the GDS vulnerability.\n");
+ return false;
+}
+
+/* Check if Key Locker is secure enough to be used. */
+static bool __init secure_keylocker(void)
+{
+ if (boot_cpu_has_bug(X86_BUG_GDS) && !have_gds_mitigation())
+ return false;
+
+ return true;
+}
+
static int __init init_keylocker(void)
{
u32 eax, ebx, ecx, edx;
@@ -125,6 +157,9 @@ static int __init init_keylocker(void)
goto clear_cap;
}
+ if (!secure_keylocker())
+ goto clear_cap;
+
cr4_set_bits(X86_CR4_KEYLOCKER);
/* AESKLE depends on CR4.KEYLOCKER */
--
2.34.1
Key Locker is a CPU feature to reduce key exfiltration opportunities.
It converts the AES key into an encoded form, called 'key handle', to
reduce the exposure of private key material in memory.
This key conversion as well as all subsequent data transformations are
provided by new AES instructions ('AES-KL'). AES-KL is analogous to
that of AES-NI as maintains a similar programming interface.
Support the XTS mode as the primary use case is dm-crypt. The support has
some details worth mentioning, which differentiate itself from AES-NI,
that users may need to be aware of:
== Key Handle Restriction ==
The AES-KL instruction set supports selecting key usage restrictions at
key handle creation time. Restrict all key handles created by the kernel
to kernel mode use only.
The AES-KL instructions themselves are executable in userspace. This
restriction enforces the mode consistency in its operation.
If the key handle is created in userspace but referenced in the kernel,
then encrypt() and decrypt() functions will return -EINVAL.
=== AES-NI Dependency for AES Compliance ===
Key Locker is not AES compliant as it lacks 192-bit key support.
However, per the expectations of Linux crypto-cipher implementations,
the software cipher implementation must support all the AES-compliant
key sizes.
The AES-KL cipher implementation achieves this constraint by logging a
warning and falling back to AES-NI. In other words, the 192-bit
key-size limitation for what can be converted into a key handle is
only documented, not enforced.
This then creates a rather strong dependency on AES-NI. If this driver
supports a module build, the exported AES-NI functions cannot be inlined.
More importantly, indirect calls will impact the performance.
To simplify, disallow a module build for AES-KL and always select AES-NI.
This restriction can be relaxed if strong use cases arise against it.
== Wrapping Key Restore Failure Handling ==
In the event of hardware failure, the wrapping key is lost from deep
sleep states. Then, the wrapping key turns to zero which is an unusable
state.
The x86 core provides valid_keylocker() to indicate the failure.
Subsequent setkey() as well as encode()/decode() can check it and return
-ENODEV if failed. In this way, an error code can be returned, instead of
facing abrupt exceptions.
== Userspace Exposition ==
The Keylocker implementations so far have measurable performance
penalties. So, keep AES-NI as the default.
However, with a slow storage device, storage bandwidth is the bottleneck,
even if disk encryption is enabled by AES-KL. Thus, it is an end-user
consideration for selecting AES-KL. Users may pick it according to the
name 'xts-aes-aeskl' shown in /proc/crypto.
== 64-bit Only ==
Support 64-bit only, as the 32-bit kernel is being deprecated.
Signed-off-by: Chang S. Bae <[email protected]>
Acked-by: Dan Williams <[email protected]>
Cc: Eric Biggers <[email protected]>
Cc: Ard Biesheuvel <[email protected]>
Cc: Herbert Xu <[email protected]>
---
Changes from v8:
* Rebase on the upstream changes.
* Combine the XTS enc/dec assembly code in a macro. (Eric Biggers)
* Define setkey() as void instead of returning 'int'. (Eric Biggers)
* Rearrange the assembly code to reduce jumps especially for success
cases. (Eric Biggers)
* Update the changelog for clarification. (Eric Biggers)
* Exclude module build.
Changes from v7:
* Update the changelog -- remove 'API Limitation'. (Eric Biggers)
* Update the comment for valid_keylocker(). (Eric Biggers)
* Improve the code:
- Remove the key-length check and simplify the code. (Eric Biggers)
- Remove aeskl_dec() and __aeskl_dec() as not needed.
- Simplify the register-function return handling. (Eric Biggers)
- Rename setkey functions for coherent naming:
aeskl_setkey() -> __aeskl_setkey(),
aeskl_setkey_common() -> aeskl_setkey(),
aeskl_xts_setkey() -> xts_setkey()
- Revert an unnecessary comment.
Changes from v6:
* Merge all the AES-KL patches. (Eric Biggers)
* Make the driver for the 64-bit mode only. (Eric Biggers)
* Rework the key-size check code:
- Trim unnecessary checks. (Eric Biggers)
- Document the reason
- Make sure both XTS keys with the same size
* Adjust the Kconfig change:
- Move the location. (Robert Elliott)
- Trim the description to follow others such as AES-NI.
* Update the changelog:
- Explain the priority value for the common name under 'User
Exposition' (renamed from 'Performance'). (Eric Biggers)
- Trim the introduction
- Switch to more imperative mood for those explaining the code
change
- Add a new section '64-bit Only'
* Adjust the ASM code to return a proper error code. (Eric Biggers)
* Update assembly code macros:
- Remove unused one.
- Document the reason for the duplicated ones.
Changes from v5:
* Replace the ret instruction with RET as rebased on the upstream -- commit
f94909ceb1ed ("x86: Prepare asm files for straight-line-speculation").
Changes from v3:
* Exclude non-AES-KL objects. (Eric Biggers)
* Simplify the assembler dependency check. (Peter Zijlstra)
* Trim the Kconfig help text. (Dan Williams)
* Fix a defined-but-not-used warning.
Changes from RFC v2:
* Move out each mode support in new patches.
* Update the changelog to describe the limitation and the tradeoff
clearly. (Andy Lutomirski)
Changes from RFC v1:
* Rebased on the refactored code. (Ard Biesheuvel)
* Dropped exporting the single block interface. (Ard Biesheuvel)
* Fixed the fallback and error handling paths. (Ard Biesheuvel)
* Revised the module description. (Dave Hansen and Peter Zijlstra)
* Made the build depend on the binutils version to support new
instructions. (Borislav Petkov and Peter Zijlstra)
* Updated the changelog accordingly.
---
arch/x86/Kconfig.assembler | 5 +
arch/x86/crypto/Kconfig | 17 ++
arch/x86/crypto/Makefile | 3 +
arch/x86/crypto/aes-helper_glue.h | 7 +-
arch/x86/crypto/aeskl-intel_asm.S | 412 +++++++++++++++++++++++++++++
arch/x86/crypto/aeskl-intel_glue.c | 187 +++++++++++++
arch/x86/crypto/aeskl-intel_glue.h | 35 +++
arch/x86/crypto/aesni-intel_glue.c | 30 +--
arch/x86/crypto/aesni-intel_glue.h | 40 +++
9 files changed, 704 insertions(+), 32 deletions(-)
create mode 100644 arch/x86/crypto/aeskl-intel_asm.S
create mode 100644 arch/x86/crypto/aeskl-intel_glue.c
create mode 100644 arch/x86/crypto/aeskl-intel_glue.h
create mode 100644 arch/x86/crypto/aesni-intel_glue.h
diff --git a/arch/x86/Kconfig.assembler b/arch/x86/Kconfig.assembler
index 8ad41da301e5..0e58f2b61dd3 100644
--- a/arch/x86/Kconfig.assembler
+++ b/arch/x86/Kconfig.assembler
@@ -25,6 +25,11 @@ config AS_GFNI
help
Supported by binutils >= 2.30 and LLVM integrated assembler
+config AS_HAS_KEYLOCKER
+ def_bool $(as-instr,encodekey256 %eax$(comma)%eax)
+ help
+ Supported by binutils >= 2.36 and LLVM integrated assembler >= V12
+
config AS_WRUSS
def_bool $(as-instr,wrussq %rax$(comma)(%rbx))
help
diff --git a/arch/x86/crypto/Kconfig b/arch/x86/crypto/Kconfig
index c9e59589a1ce..067bb149998b 100644
--- a/arch/x86/crypto/Kconfig
+++ b/arch/x86/crypto/Kconfig
@@ -29,6 +29,23 @@ config CRYPTO_AES_NI_INTEL
Architecture: x86 (32-bit and 64-bit) using:
- AES-NI (AES new instructions)
+config CRYPTO_AES_KL
+ bool "Ciphers: AES, modes: XTS (AES-KL)"
+ depends on X86 && 64BIT
+ depends on AS_HAS_KEYLOCKER
+ select CRYPTO_AES_NI_INTEL
+ select X86_KEYLOCKER
+
+ help
+ Block cipher: AES cipher algorithms
+ Length-preserving ciphers: AES with XTS
+
+ Architecture: x86 (64-bit) using:
+ - AES-KL (AES Key Locker)
+ - AES-NI for a 192-bit key
+
+ See Documentation/arch/x86/keylocker.rst for more details.
+
config CRYPTO_BLOWFISH_X86_64
tristate "Ciphers: Blowfish, modes: ECB, CBC"
depends on X86 && 64BIT
diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile
index 9aa46093c91b..ae2aa7abd151 100644
--- a/arch/x86/crypto/Makefile
+++ b/arch/x86/crypto/Makefile
@@ -50,6 +50,9 @@ obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
+obj-$(CONFIG_CRYPTO_AES_KL) += aeskl-intel.o
+aeskl-intel-y := aeskl-intel_asm.o aeskl-intel_glue.o
+
obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o
sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o
sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o
diff --git a/arch/x86/crypto/aes-helper_glue.h b/arch/x86/crypto/aes-helper_glue.h
index 52ba1fe5cf71..262c1cec0011 100644
--- a/arch/x86/crypto/aes-helper_glue.h
+++ b/arch/x86/crypto/aes-helper_glue.h
@@ -19,16 +19,17 @@
#include <crypto/internal/aead.h>
#include <crypto/internal/simd.h>
+#include "aeskl-intel_glue.h"
+
#define AES_ALIGN 16
#define AES_ALIGN_ATTR __attribute__((__aligned__(AES_ALIGN)))
#define AES_ALIGN_EXTRA ((AES_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1))
#define XTS_AES_CTX_SIZE (sizeof(struct aes_xts_ctx) + AES_ALIGN_EXTRA)
-/*
- * Preserve data types for various AES implementations available in x86
- */
+/* Data types for the two AES implementations available in x86 */
union x86_aes_ctx {
struct crypto_aes_ctx aesni;
+ struct aeskl_ctx aeskl;
};
struct aes_xts_ctx {
diff --git a/arch/x86/crypto/aeskl-intel_asm.S b/arch/x86/crypto/aeskl-intel_asm.S
new file mode 100644
index 000000000000..81af7f61aab5
--- /dev/null
+++ b/arch/x86/crypto/aeskl-intel_asm.S
@@ -0,0 +1,412 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Implement AES algorithm using AES Key Locker instructions.
+ *
+ * Most code is based from the AES-NI implementation, aesni-intel_asm.S
+ *
+ */
+
+#include <linux/linkage.h>
+#include <linux/cfi_types.h>
+#include <asm/errno.h>
+#include <asm/inst.h>
+#include <asm/frame.h>
+#include "aes-helper_asm.S"
+
+.text
+
+#define STATE1 %xmm0
+#define STATE2 %xmm1
+#define STATE3 %xmm2
+#define STATE4 %xmm3
+#define STATE5 %xmm4
+#define STATE6 %xmm5
+#define STATE7 %xmm6
+#define STATE8 %xmm7
+#define STATE STATE1
+
+#define IV %xmm9
+#define KEY %xmm10
+#define INC %xmm13
+
+#define IN %xmm8
+
+#define HANDLEP %rdi
+#define OUTP %rsi
+#define KLEN %r9d
+#define INP %rdx
+#define T1 %r10
+#define LEN %rcx
+#define IVP %r8
+
+#define UKEYP OUTP
+#define GF128MUL_MASK %xmm11
+
+/*
+ * void __aeskl_setkey(struct crypto_aes_ctx *handlep, const u8 *ukeyp,
+ * unsigned int key_len)
+ */
+SYM_FUNC_START(__aeskl_setkey)
+ FRAME_BEGIN
+ movl %edx, 480(HANDLEP)
+ movdqu (UKEYP), STATE1
+ mov $1, %eax
+ cmp $16, %dl
+ je .Lsetkey_128
+
+ movdqu 0x10(UKEYP), STATE2
+ encodekey256 %eax, %eax
+ movdqu STATE4, 0x30(HANDLEP)
+ jmp .Lsetkey_end
+.Lsetkey_128:
+ encodekey128 %eax, %eax
+
+.Lsetkey_end:
+ movdqu STATE1, (HANDLEP)
+ movdqu STATE2, 0x10(HANDLEP)
+ movdqu STATE3, 0x20(HANDLEP)
+
+ FRAME_END
+ RET
+SYM_FUNC_END(__aeskl_setkey)
+
+/*
+ * int __aeskl_enc(const void *handlep, u8 *outp, const u8 *inp)
+ */
+SYM_FUNC_START(__aeskl_enc)
+ FRAME_BEGIN
+ movdqu (INP), STATE
+ movl 480(HANDLEP), KLEN
+
+ cmp $16, KLEN
+ je .Lenc_128
+ aesenc256kl (HANDLEP), STATE
+ jz .Lenc_err
+ xor %rax, %rax
+ jmp .Lenc_end
+.Lenc_128:
+ aesenc128kl (HANDLEP), STATE
+ jz .Lenc_err
+ xor %rax, %rax
+ jmp .Lenc_end
+
+.Lenc_err:
+ mov $(-EINVAL), %rax
+.Lenc_end:
+ movdqu STATE, (OUTP)
+ FRAME_END
+ RET
+SYM_FUNC_END(__aeskl_enc)
+
+/*
+ * XTS implementation
+ */
+
+/*
+ * _aeskl_gf128mul_x_ble: internal ABI
+ * Multiply in GF(2^128) for XTS IVs
+ * input:
+ * IV: current IV
+ * GF128MUL_MASK == mask with 0x87 and 0x01
+ * output:
+ * IV: next IV
+ * changed:
+ * CTR: == temporary value
+ *
+ * While based on the AES-NI code, this macro is separated here due to
+ * the register constraint. E.g., aesencwide256kl has implicit
+ * operands: XMM0-7.
+ */
+#define _aeskl_gf128mul_x_ble() \
+ pshufd $0x13, IV, KEY; \
+ paddq IV, IV; \
+ psrad $31, KEY; \
+ pand GF128MUL_MASK, KEY; \
+ pxor KEY, IV;
+
+.macro XTS_ENC_DEC operation
+ FRAME_BEGIN
+ movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK
+ movups (IVP), IV
+
+ mov 480(HANDLEP), KLEN
+
+.ifc \operation, dec
+ test $15, LEN
+ jz .Lxts_op8_\@
+ sub $16, LEN
+.endif
+
+.Lxts_op8_\@:
+ sub $128, LEN
+ jl .Lxts_op1_pre_\@
+
+ movdqa IV, STATE1
+ movdqu (INP), INC
+ pxor INC, STATE1
+ movdqu IV, (OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE2
+ movdqu 0x10(INP), INC
+ pxor INC, STATE2
+ movdqu IV, 0x10(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE3
+ movdqu 0x20(INP), INC
+ pxor INC, STATE3
+ movdqu IV, 0x20(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE4
+ movdqu 0x30(INP), INC
+ pxor INC, STATE4
+ movdqu IV, 0x30(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE5
+ movdqu 0x40(INP), INC
+ pxor INC, STATE5
+ movdqu IV, 0x40(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE6
+ movdqu 0x50(INP), INC
+ pxor INC, STATE6
+ movdqu IV, 0x50(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE7
+ movdqu 0x60(INP), INC
+ pxor INC, STATE7
+ movdqu IV, 0x60(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+ movdqa IV, STATE8
+ movdqu 0x70(INP), INC
+ pxor INC, STATE8
+ movdqu IV, 0x70(OUTP)
+
+ cmp $16, KLEN
+ je .Lxts_op8_128_\@
+.ifc \operation, dec
+ aesdecwide256kl (%rdi)
+.else
+ aesencwide256kl (%rdi)
+.endif
+ jz .Lxts_op_err_\@
+ jmp .Lxts_op8_end_\@
+.Lxts_op8_128_\@:
+.ifc \operation, dec
+ aesdecwide128kl (%rdi)
+.else
+ aesencwide128kl (%rdi)
+.endif
+ jz .Lxts_op_err_\@
+
+.Lxts_op8_end_\@:
+ movdqu 0x00(OUTP), INC
+ pxor INC, STATE1
+ movdqu STATE1, 0x00(OUTP)
+
+ movdqu 0x10(OUTP), INC
+ pxor INC, STATE2
+ movdqu STATE2, 0x10(OUTP)
+
+ movdqu 0x20(OUTP), INC
+ pxor INC, STATE3
+ movdqu STATE3, 0x20(OUTP)
+
+ movdqu 0x30(OUTP), INC
+ pxor INC, STATE4
+ movdqu STATE4, 0x30(OUTP)
+
+ movdqu 0x40(OUTP), INC
+ pxor INC, STATE5
+ movdqu STATE5, 0x40(OUTP)
+
+ movdqu 0x50(OUTP), INC
+ pxor INC, STATE6
+ movdqu STATE6, 0x50(OUTP)
+
+ movdqu 0x60(OUTP), INC
+ pxor INC, STATE7
+ movdqu STATE7, 0x60(OUTP)
+
+ movdqu 0x70(OUTP), INC
+ pxor INC, STATE8
+ movdqu STATE8, 0x70(OUTP)
+
+ _aeskl_gf128mul_x_ble()
+
+ add $128, INP
+ add $128, OUTP
+ test LEN, LEN
+ jnz .Lxts_op8_\@
+
+.Lxts_op_ret_\@:
+ movups IV, (IVP)
+ xor %rax, %rax
+ FRAME_END
+ RET
+
+.Lxts_op1_pre_\@:
+ add $128, LEN
+ jz .Lxts_op_ret_\@
+.ifc \operation, enc
+ sub $16, LEN
+ jl .Lxts_op_cts4_\@
+.endif
+
+.Lxts_op1_\@:
+ movdqu (INP), STATE1
+
+.ifc \operation, dec
+ add $16, INP
+ sub $16, LEN
+ jl .Lxts_op_cts1_\@
+.endif
+
+ pxor IV, STATE1
+
+ cmp $16, KLEN
+ je .Lxts_op1_128_\@
+.ifc \operation, dec
+ aesdec256kl (HANDLEP), STATE1
+.else
+ aesenc256kl (HANDLEP), STATE1
+.endif
+ jz .Lxts_op_err_\@
+ jmp .Lxts_op1_end_\@
+.Lxts_op1_128_\@:
+.ifc \operation, dec
+ aesdec128kl (HANDLEP), STATE1
+.else
+ aesenc128kl (HANDLEP), STATE1
+.endif
+ jz .Lxts_op_err_\@
+
+.Lxts_op1_end_\@:
+ pxor IV, STATE1
+ _aeskl_gf128mul_x_ble()
+
+ test LEN, LEN
+ jz .Lxts_op1_out_\@
+
+.ifc \operation, enc
+ add $16, INP
+ sub $16, LEN
+ jl .Lxts_op_cts1_\@
+.endif
+
+ movdqu STATE1, (OUTP)
+ add $16, OUTP
+ jmp .Lxts_op1_\@
+
+.Lxts_op1_out_\@:
+ movdqu STATE1, (OUTP)
+ jmp .Lxts_op_ret_\@
+
+.Lxts_op_cts4_\@:
+.ifc \operation, enc
+ movdqu STATE8, STATE1
+ sub $16, OUTP
+.endif
+
+.Lxts_op_cts1_\@:
+.ifc \operation, dec
+ movdqa IV, STATE5
+ _aeskl_gf128mul_x_ble()
+
+ pxor IV, STATE1
+
+ cmp $16, KLEN
+ je .Lxts_dec1_cts_pre_128_\@
+ aesdec256kl (HANDLEP), STATE1
+ jz .Lxts_op_err_\@
+ jmp .Lxts_dec1_cts_pre_end_\@
+.Lxts_dec1_cts_pre_128_\@:
+ aesdec128kl (HANDLEP), STATE1
+ jz .Lxts_op_err_\@
+.Lxts_dec1_cts_pre_end_\@:
+ pxor IV, STATE1
+.endif
+
+ lea .Lcts_permute_table(%rip), T1
+ add LEN, INP /* rewind input pointer */
+ add $16, LEN /* # bytes in final block */
+ movups (INP), IN
+
+ mov T1, IVP
+ add $32, IVP
+ add LEN, T1
+ sub LEN, IVP
+ add OUTP, LEN
+
+ movups (T1), STATE2
+ movaps STATE1, STATE3
+ pshufb STATE2, STATE1
+ movups STATE1, (LEN)
+
+ movups (IVP), STATE1
+ pshufb STATE1, IN
+ pblendvb STATE3, IN
+ movaps IN, STATE1
+
+.ifc \operation, dec
+ pxor STATE5, STATE1
+.else
+ pxor IV, STATE1
+.endif
+
+ cmp $16, KLEN
+ je .Lxts_op1_cts_128_\@
+.ifc \operation, dec
+ aesdec256kl (HANDLEP), STATE1
+.else
+ aesenc256kl (HANDLEP), STATE1
+.endif
+ jz .Lxts_op_err_\@
+ jmp .Lxts_op1_cts_end_\@
+.Lxts_op1_cts_128_\@:
+.ifc \operation, dec
+ aesdec128kl (HANDLEP), STATE1
+.else
+ aesenc128kl (HANDLEP), STATE1
+.endif
+ jz .Lxts_op_err_\@
+
+.Lxts_op1_cts_end_\@:
+.ifc \operation, dec
+ pxor STATE5, STATE1
+.else
+ pxor IV, STATE1
+.endif
+ movups STATE1, (OUTP)
+ xor %rax, %rax
+ FRAME_END
+ RET
+
+.Lxts_op_err_\@:
+ mov $(-EINVAL), %rax
+ FRAME_END
+ RET
+.endm
+
+/*
+ * int __aeskl_xts_encrypt(const struct aeskl_ctx *handlep, u8 *outp,
+ * const u8 *inp, unsigned int klen, le128 *ivp)
+ */
+SYM_FUNC_START(__aeskl_xts_encrypt)
+ XTS_ENC_DEC enc
+SYM_FUNC_END(__aeskl_xts_encrypt)
+
+/*
+ * int __aeskl_xts_decrypt(const struct crypto_aes_ctx *handlep, u8 *outp,
+ * const u8 *inp, unsigned int klen, le128 *ivp)
+ */
+SYM_FUNC_START(__aeskl_xts_decrypt)
+ XTS_ENC_DEC dec
+SYM_FUNC_END(__aeskl_xts_decrypt)
+
diff --git a/arch/x86/crypto/aeskl-intel_glue.c b/arch/x86/crypto/aeskl-intel_glue.c
new file mode 100644
index 000000000000..7672c4836da8
--- /dev/null
+++ b/arch/x86/crypto/aeskl-intel_glue.c
@@ -0,0 +1,187 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Support for AES Key Locker instructions. This file contains glue
+ * code and the real AES implementation is in aeskl-intel_asm.S.
+ *
+ * Most code is based on AES-NI glue code, aesni-intel_glue.c
+ */
+
+#include <linux/types.h>
+#include <linux/module.h>
+#include <linux/err.h>
+#include <crypto/algapi.h>
+#include <crypto/aes.h>
+#include <crypto/xts.h>
+#include <crypto/internal/skcipher.h>
+#include <crypto/internal/simd.h>
+#include <asm/simd.h>
+#include <asm/cpu_device_id.h>
+#include <asm/fpu/api.h>
+#include <asm/keylocker.h>
+
+#include "aes-helper_glue.h"
+#include "aesni-intel_glue.h"
+
+asmlinkage void __aeskl_setkey(struct aeskl_ctx *ctx, const u8 *in_key, unsigned int keylen);
+
+asmlinkage int __aeskl_enc(const void *ctx, u8 *out, const u8 *in);
+
+asmlinkage int __aeskl_xts_encrypt(const struct aeskl_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv);
+asmlinkage int __aeskl_xts_decrypt(const struct aeskl_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv);
+
+/*
+ * If a hardware failure occurs, the wrapping key may be lost during
+ * sleep states. The state of the feature can be retrieved via
+ * valid_keylocker().
+ *
+ * Since disabling can occur preemptively, check for availability on
+ * every use along with kernel_fpu_begin().
+ */
+
+static int aeskl_setkey(union x86_aes_ctx *ctx, const u8 *in_key, unsigned int keylen)
+{
+ int err;
+
+ if (!crypto_simd_usable())
+ return -EBUSY;
+
+ err = aes_check_keylen(keylen);
+ if (err)
+ return err;
+
+ if (unlikely(keylen == AES_KEYSIZE_192)) {
+ pr_warn_once("AES-KL does not support 192-bit key. Use AES-NI.\n");
+ kernel_fpu_begin();
+ aesni_set_key(&ctx->aesni, in_key, keylen);
+ kernel_fpu_end();
+ return 0;
+ }
+
+ if (!valid_keylocker())
+ return -ENODEV;
+
+ kernel_fpu_begin();
+ __aeskl_setkey(&ctx->aeskl, in_key, keylen);
+ kernel_fpu_end();
+ return 0;
+}
+
+static inline int aeskl_enc(const void *ctx, u8 *out, const u8 *in)
+{
+ if (!valid_keylocker())
+ return -ENODEV;
+
+ return __aeskl_enc(ctx, out, in);
+}
+
+static inline int aeskl_xts_encrypt(const union x86_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv)
+{
+ if (!valid_keylocker())
+ return -ENODEV;
+
+ return __aeskl_xts_encrypt(&ctx->aeskl, out, in, len, iv);
+}
+
+static inline int aeskl_xts_decrypt(const union x86_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv)
+{
+ if (!valid_keylocker())
+ return -ENODEV;
+
+ return __aeskl_xts_decrypt(&ctx->aeskl, out, in, len, iv);
+}
+
+static int xts_setkey(struct crypto_skcipher *tfm, const u8 *key,
+ unsigned int keylen)
+{
+ return xts_setkey_common(tfm, key, keylen, aeskl_setkey);
+}
+
+static inline u32 xts_keylen(struct skcipher_request *req)
+{
+ struct aes_xts_ctx *ctx = aes_xts_ctx(crypto_skcipher_reqtfm(req));
+
+ return ctx->crypt_ctx.aeskl.key_length;
+}
+
+static int xts_encrypt(struct skcipher_request *req)
+{
+ u32 keylen = xts_keylen(req);
+
+ if (likely(keylen != AES_KEYSIZE_192))
+ return xts_crypt_common(req, aeskl_xts_encrypt, aeskl_enc);
+ else
+ return xts_crypt_common(req, aesni_xts_encrypt, aesni_enc);
+}
+
+static int xts_decrypt(struct skcipher_request *req)
+{
+ u32 keylen = xts_keylen(req);
+
+ if (likely(keylen != AES_KEYSIZE_192))
+ return xts_crypt_common(req, aeskl_xts_decrypt, aeskl_enc);
+ else
+ return xts_crypt_common(req, aesni_xts_decrypt, aesni_enc);
+}
+
+static struct skcipher_alg aeskl_skciphers[] = {
+ {
+ .base = {
+ .cra_name = "__xts(aes)",
+ .cra_driver_name = "__xts-aes-aeskl",
+ .cra_priority = 200,
+ .cra_flags = CRYPTO_ALG_INTERNAL,
+ .cra_blocksize = AES_BLOCK_SIZE,
+ .cra_ctxsize = XTS_AES_CTX_SIZE,
+ .cra_module = THIS_MODULE,
+ },
+ .min_keysize = 2 * AES_MIN_KEY_SIZE,
+ .max_keysize = 2 * AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .walksize = 2 * AES_BLOCK_SIZE,
+ .setkey = xts_setkey,
+ .encrypt = xts_encrypt,
+ .decrypt = xts_decrypt,
+ }
+};
+
+static struct simd_skcipher_alg *aeskl_simd_skciphers[ARRAY_SIZE(aeskl_skciphers)];
+
+static int __init aeskl_init(void)
+{
+ u32 eax, ebx, ecx, edx;
+
+ if (!valid_keylocker())
+ return -ENODEV;
+
+ cpuid_count(KEYLOCKER_CPUID, 0, &eax, &ebx, &ecx, &edx);
+ if (!(ebx & KEYLOCKER_CPUID_EBX_WIDE))
+ return -ENODEV;
+
+ /*
+ * AES-KL itself does not rely on AES-NI. But, AES-KL does not
+ * support 192-bit keys. To ensure AES compliance, AES-KL falls
+ * back to AES-NI.
+ */
+ if (!boot_cpu_has(X86_FEATURE_AES))
+ return -ENODEV;
+
+ return simd_register_skciphers_compat(aeskl_skciphers, ARRAY_SIZE(aeskl_skciphers),
+ aeskl_simd_skciphers);
+}
+
+static void __exit aeskl_exit(void)
+{
+ simd_unregister_skciphers(aeskl_skciphers, ARRAY_SIZE(aeskl_skciphers),
+ aeskl_simd_skciphers);
+}
+
+late_initcall(aeskl_init);
+module_exit(aeskl_exit);
+
+MODULE_DESCRIPTION("Rijndael (AES) Cipher Algorithm, AES Key Locker implementation");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_CRYPTO("aes");
diff --git a/arch/x86/crypto/aeskl-intel_glue.h b/arch/x86/crypto/aeskl-intel_glue.h
new file mode 100644
index 000000000000..57cfd6c55a4f
--- /dev/null
+++ b/arch/x86/crypto/aeskl-intel_glue.h
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef _AESKL_INTEL_GLUE_H
+#define _AESKL_INTEL_GLUE_H
+
+#include <crypto/aes.h>
+#include <linux/types.h>
+
+#define AESKL_AAD_SIZE 16
+#define AESKL_TAG_SIZE 16
+#define AESKL_CIPHERTEXT_MAX AES_KEYSIZE_256
+
+/* The Key Locker handle is an encoded form of the AES key. */
+struct aeskl_handle {
+ u8 additional_authdata[AESKL_AAD_SIZE];
+ u8 integrity_tag[AESKL_TAG_SIZE];
+ u8 ciphre_text[AESKL_CIPHERTEXT_MAX];
+};
+
+/*
+ * Key Locker does not support 192-bit key size. The driver needs to
+ * retrieve the key size in the first place. The offset of the
+ * 'key_length' field here should be compatible with struct
+ * crypto_aes_ctx.
+ */
+#define AESKL_CTX_RESERVED (sizeof(struct crypto_aes_ctx) - sizeof(struct aeskl_handle) \
+ - sizeof(u32))
+
+struct aeskl_ctx {
+ struct aeskl_handle handle;
+ u8 reserved[AESKL_CTX_RESERVED];
+ u32 key_length;
+};
+
+#endif /* _AESKL_INTEL_GLUE_H */
diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c
index 4ac7b9a28967..d9c4aa055383 100644
--- a/arch/x86/crypto/aesni-intel_glue.c
+++ b/arch/x86/crypto/aesni-intel_glue.c
@@ -37,6 +37,7 @@
#include <linux/static_call.h>
#include "aes-helper_glue.h"
+#include "aesni-intel_glue.h"
#define RFC4106_HASH_SUBKEY_SIZE 16
#define AES_BLOCK_MASK (~(AES_BLOCK_SIZE - 1))
@@ -72,9 +73,6 @@ struct gcm_context_data {
u8 hash_keys[GCM_BLOCK_LEN * 16];
};
-asmlinkage void aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
- unsigned int key_len);
-asmlinkage void __aesni_enc(const void *ctx, u8 *out, const u8 *in);
asmlinkage void __aesni_dec(const void *ctx, u8 *out, const u8 *in);
asmlinkage void aesni_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out,
const u8 *in, unsigned int len);
@@ -89,21 +87,9 @@ asmlinkage void aesni_cts_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out,
asmlinkage void aesni_cts_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out,
const u8 *in, unsigned int len, u8 *iv);
-static inline int aesni_enc(const void *ctx, u8 *out, const u8 *in)
-{
- __aesni_enc(ctx, out, in);
- return 0;
-}
-
#define AVX_GEN2_OPTSIZE 640
#define AVX_GEN4_OPTSIZE 4096
-asmlinkage void __aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out,
- const u8 *in, unsigned int len, u8 *iv);
-
-asmlinkage void __aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out,
- const u8 *in, unsigned int len, u8 *iv);
-
#ifdef CONFIG_X86_64
asmlinkage void aesni_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out,
@@ -271,20 +257,6 @@ static inline int aesni_xts_setkey(union x86_aes_ctx *ctx,
return aes_set_key_common(&ctx->aesni, in_key, key_len);
}
-static inline int aesni_xts_encrypt(const union x86_aes_ctx *ctx, u8 *out, const u8 *in,
- unsigned int len, u8 *iv)
-{
- __aesni_xts_encrypt(&ctx->aesni, out, in, len, iv);
- return 0;
-}
-
-static inline int aesni_xts_decrypt(const union x86_aes_ctx *ctx, u8 *out, const u8 *in,
- unsigned int len, u8 *iv)
-{
- __aesni_xts_decrypt(&ctx->aesni, out, in, len, iv);
- return 0;
-}
-
static int aesni_skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key,
unsigned int len)
{
diff --git a/arch/x86/crypto/aesni-intel_glue.h b/arch/x86/crypto/aesni-intel_glue.h
new file mode 100644
index 000000000000..999f81f5bcde
--- /dev/null
+++ b/arch/x86/crypto/aesni-intel_glue.h
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * These are AES-NI functions that are used by the AES-KL code as a
+ * fallback when it is given a 192-bit key. Key Locker does not support
+ * 192-bit keys.
+ */
+
+#ifndef _AESNI_INTEL_GLUE_H
+#define _AESNI_INTEL_GLUE_H
+
+asmlinkage void aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
+ unsigned int key_len);
+asmlinkage void __aesni_enc(const void *ctx, u8 *out, const u8 *in);
+asmlinkage void __aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out,
+ const u8 *in, unsigned int len, u8 *iv);
+asmlinkage void __aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out,
+ const u8 *in, unsigned int len, u8 *iv);
+
+static inline int aesni_enc(const void *ctx, u8 *out, const u8 *in)
+{
+ __aesni_enc(ctx, out, in);
+ return 0;
+}
+
+static inline int aesni_xts_encrypt(const union x86_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv)
+{
+ __aesni_xts_encrypt(&ctx->aesni, out, in, len, iv);
+ return 0;
+}
+
+static inline int aesni_xts_decrypt(const union x86_aes_ctx *ctx, u8 *out, const u8 *in,
+ unsigned int len, u8 *iv)
+{
+ __aesni_xts_decrypt(&ctx->aesni, out, in, len, iv);
+ return 0;
+}
+
+#endif /* _AESNI_INTEL_GLUE_H */
--
2.34.1
On 4/15/2024 3:54 PM, Eric Biggers wrote:
>
> Sure, you could still use VEX-coded 128-bit instructions for everything other
> than the actual AES (for example, the XTS tweak computation) though, right?
> They're a bit more convenient to work with since they are non-destructive.
Right.
Thanks,
Chang