The AMD Zen4 core supports a new feature called Automatic IBRS
(Indirect Branch Restricted Speculation).
Enable Automatic IBRS by default if the CPU feature is present.
It typically provides greater performance over the incumbent
generic retpolines mitigation.
Patches 1-3 take the existing CPUID 0x80000021 EAX feature bits
that are being propagated to the guest and define scattered
versions for patch 4.
Patch 4 moves CPUID 0x80000021 EAX feature bits propagation code
to kvm_set_cpu_caps().
Patch 5 Defines the AutoIBRS feature bit.
Patch 6 Adds support for AutoIBRS by turning its EFER
enablement bit on at startup if the feature is available.
Patch 7 Adds support for propagating AutoIBRS to the guest.
Thanks to Babu Moger for helping debug guest propagation (patch 7).
Also thanks to Boris and Paolo for their early version diffs.
v5: Address v4 comments from Dave Hansen, Pawan Gupta, and Boris:
- Don't add new user-visible 'autoibrs' command line
options that we have to document: reuse 'eibrs'
- Update Documentation/admin-guide/hw-vuln/spectre.rst
- Add NO_EIBRS_PBRSB to Hygon as well
- Re-word commit texts to not use words like 'us'
v4: https://lore.kernel.org/lkml/[email protected]/
Moved some kvm bits that had crept into patch 6/7 back into 7/7,
and addressed v3 comments:
- Don't put ", kvm" in titles of patches that don't touch kvm. [SeanC]
- () after function names, i.e. kvm_set_cpu_caps(). [SeanC]
- follow the established kvm_cpu_cap_init_scattered() style [SeanC]
- Add using cpu_feature_enabled() instead of static_cpu_has() to
commit text [SeanC]
- Pawan Gupta mentioned that the ordering of enabling the Intel
feature bit past Intel EIBRS bug detection could be avoided
by setting NO_EIBRS_PBRSB to cpu_vuln_whitelist, so did that
which allowed regrouping all EIBRS related code to one place
in cpu_set_bug_bits().
v3: https://lore.kernel.org/lkml/[email protected]/
- Remove Co-developed-bys. They require signed-off-bys,
so co-developers need to add them themselves.
- update check_null_seg_clears_base() [Boris]
- Made the feature bit additions separate patches
because v2 patch was clearly doing too many things at once.
v2: https://lore.kernel.org/lkml/[email protected]/
https://lkml.org/lkml/2022/11/23/1690
- Use synthetic/scattered bits instead of introducing new leaf [Boris]
- Combine the rest of the leaf's bits being used [Paolo]
Note: Bits not used by the host can be moved to kvm/cpuid.c if
maintainers do not want them in cpufeatures.h.
- Hoist bitsetting code to kvm_set_cpu_caps(), and use
cpuid_entry_override() in __do_cpuid_func() [Paolo]
- Reuse SPECTRE_V2_EIBRS spectre_v2_mitigation enum [Boris, PeterZ, D.Hansen]
- Change from Boris' diff:
Moved setting X86_FEATURE_IBRS_ENHANCED to after BUG_EIBRS_PBRSB
so PBRSB mitigations wouldn't be enabled.
- Allow for users to specify "autoibrs,lfence/retpoline" instead
of actively preventing the extra protections. AutoIBRS doesn't
require the extra protection, but we allow it anyway.
v1: https://lore.kernel.org/lkml/[email protected]/
Signed-off-by: Kim Phillips <[email protected]>
Cc: Babu Moger <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Boris Ostrovsky <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: "H. Peter Anvin" <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Joao Martins <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Paolo Bonzini <[email protected]>
Cc: Sean Christopherson <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: David Woodhouse <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Tony Luck <[email protected]>
Cc: Tom Lendacky <[email protected]>
Cc: Alexey Kardashevskiy <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Kim Phillips (7):
x86/cpu: Define a scattered No Nested Data Breakpoints feature bit
x86/cpu: Define a scattered Null Selector Clears Base feature bit
x86/cpu: Make X86_FEATURE_LFENCE_RDTSC a scattered feature bit
x86/cpu, kvm: Move CPUID 0x80000021 EAX feature bits propagation to
kvm_set_cpu_caps()
x86/cpu: Define a scattered AMD Automatic IBRS feature bit
x86/cpu: Support AMD Automatic IBRS
x86/cpu, kvm: Propagate the AMD Automatic IBRS feature to the guest
Documentation/admin-guide/hw-vuln/spectre.rst | 6 ++---
.../admin-guide/kernel-parameters.txt | 6 ++---
arch/x86/include/asm/cpufeatures.h | 3 +++
arch/x86/include/asm/msr-index.h | 2 ++
arch/x86/kernel/cpu/bugs.c | 20 +++++++++------
arch/x86/kernel/cpu/common.c | 22 ++++++++--------
arch/x86/kernel/cpu/scattered.c | 4 +++
arch/x86/kvm/cpuid.c | 25 ++++++++-----------
arch/x86/kvm/reverse_cpuid.h | 24 +++++++++++++-----
arch/x86/kvm/svm/svm.c | 3 +++
arch/x86/kvm/x86.c | 3 +++
11 files changed, 73 insertions(+), 45 deletions(-)
--
2.34.1
It's a part of the CPUID 0x80000021 leaf, and will be grouped
with other feature bits to being propagated via kvm_set_cpu_caps()
instead of open-coding them in __do_cpuid_func().
Signed-off-by: Kim Phillips <[email protected]>
---
arch/x86/include/asm/cpufeatures.h | 1 +
arch/x86/kernel/cpu/scattered.c | 1 +
2 files changed, 2 insertions(+)
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index 11a0e06362e4..b16fdcedc2b5 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -307,6 +307,7 @@
#define X86_FEATURE_SGX_EDECCSSA (11*32+18) /* "" SGX EDECCSSA user leaf function */
#define X86_FEATURE_CALL_DEPTH (11*32+19) /* "" Call depth tracking for RSB stuffing */
#define X86_FEATURE_MSR_TSX_CTRL (11*32+20) /* "" MSR IA32_TSX_CTRL (Intel) implemented */
+#define X86_FEATURE_NO_NESTED_DATA_BP (11*32+21) /* "" AMD No Nested Data Breakpoints */
/* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */
#define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */
diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
index f53944fb8f7f..079e253e1049 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -45,6 +45,7 @@ static const struct cpuid_bit cpuid_bits[] = {
{ X86_FEATURE_CPB, CPUID_EDX, 9, 0x80000007, 0 },
{ X86_FEATURE_PROC_FEEDBACK, CPUID_EDX, 11, 0x80000007, 0 },
{ X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 },
+ { X86_FEATURE_NO_NESTED_DATA_BP,CPUID_EAX, 0, 0x80000021, 0 },
{ X86_FEATURE_PERFMON_V2, CPUID_EAX, 0, 0x80000022, 0 },
{ X86_FEATURE_AMD_LBR_V2, CPUID_EAX, 1, 0x80000022, 0 },
{ 0, 0, 0, 0, 0 }
--
2.34.1
Add the AMD Automatic IBRS feature bit to those being
propagated to the guest, and enable the guest EFER bit.
Signed-off-by: Kim Phillips <[email protected]>
---
arch/x86/kvm/cpuid.c | 2 +-
arch/x86/kvm/reverse_cpuid.h | 2 ++
arch/x86/kvm/svm/svm.c | 3 +++
arch/x86/kvm/x86.c | 3 +++
4 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index dd0fe79521eb..0ed3ad3e9341 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -732,7 +732,7 @@ void kvm_set_cpu_caps(void)
kvm_cpu_cap_init_scattered(CPUID_8000_0021_EAX,
SF(NO_NESTED_DATA_BP) | SF(LFENCE_RDTSC) | 0 /* SmmPgCfgLock */ |
- SF(NULL_SEL_CLR_BASE) | 0 /* PrefetchCtlMsr */
+ SF(NULL_SEL_CLR_BASE) | SF(AUTOIBRS) | 0 /* PrefetchCtlMsr */
);
if (cpu_feature_enabled(X86_FEATURE_LFENCE_RDTSC))
kvm_cpu_cap_set(X86_FEATURE_LFENCE_RDTSC);
diff --git a/arch/x86/kvm/reverse_cpuid.h b/arch/x86/kvm/reverse_cpuid.h
index 184614e27d5b..0bf02c02bb0a 100644
--- a/arch/x86/kvm/reverse_cpuid.h
+++ b/arch/x86/kvm/reverse_cpuid.h
@@ -30,6 +30,7 @@ enum kvm_only_cpuid_leafs {
#define KVM_X86_FEATURE_NO_NESTED_DATA_BP KVM_X86_FEATURE(CPUID_8000_0021_EAX, 0)
#define KVM_X86_FEATURE_LFENCE_RDTSC KVM_X86_FEATURE(CPUID_8000_0021_EAX, 2)
#define KVM_X86_FEATURE_NULL_SEL_CLR_BASE KVM_X86_FEATURE(CPUID_8000_0021_EAX, 6)
+#define KVM_X86_FEATURE_AUTOIBRS KVM_X86_FEATURE(CPUID_8000_0021_EAX, 8)
struct cpuid_reg {
u32 function;
@@ -89,6 +90,7 @@ static __always_inline u32 __feature_translate(int x86_feature)
case X86_FEATURE_NO_NESTED_DATA_BP: return KVM_X86_FEATURE_NO_NESTED_DATA_BP;
case X86_FEATURE_LFENCE_RDTSC: return KVM_X86_FEATURE_LFENCE_RDTSC;
case X86_FEATURE_NULL_SEL_CLR_BASE: return KVM_X86_FEATURE_NULL_SEL_CLR_BASE;
+ case X86_FEATURE_AUTOIBRS: return KVM_X86_FEATURE_AUTOIBRS;
default: break;
}
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index ce362e88a567..1551a791cff0 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4946,6 +4946,9 @@ static __init int svm_hardware_setup(void)
tsc_aux_uret_slot = kvm_add_user_return_msr(MSR_TSC_AUX);
+ if (boot_cpu_has(X86_FEATURE_AUTOIBRS))
+ kvm_enable_efer_bits(EFER_AUTOIBRS);
+
/* Check for pause filtering support */
if (!boot_cpu_has(X86_FEATURE_PAUSEFILTER)) {
pause_filter_count = 0;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 2835bd796639..0697d444c715 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1688,6 +1688,9 @@ static int do_get_msr_feature(struct kvm_vcpu *vcpu, unsigned index, u64 *data)
static bool __kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer)
{
+ if (efer & EFER_AUTOIBRS && !guest_cpuid_has(vcpu, X86_FEATURE_AUTOIBRS))
+ return false;
+
if (efer & EFER_FFXSR && !guest_cpuid_has(vcpu, X86_FEATURE_FXSR_OPT))
return false;
--
2.34.1