The 'commit 67034bb9dd5e ("KVM: SVM: Add irqchip_split() checks before
enabling AVIC")' was introduced to fix miscellaneous boot-hang issues
when enable AVIC. This is mainly due to AVIC hardware doest not #vmexit
on write to LAPIC EOI register resulting in-kernel PIC and IOAPIC to
wait and do not inject new interrupts (e.g. PIT, RTC).
This limits AVIC to only work with kernel_irqchip=split mode, which is
not currently enabled by default, and also required user-space to
support split irqchip model, which might not be the case.
The goal of this series is to enable AVIC to work in both irqchip modes,
by allowing AVIC to be deactivated temporarily during runtime, and fallback
to legacy interrupt injection mode (w/ vINTR and interrupt windows)
when needed, and then re-enabled subsequently (a.k.a Dynamic APICv).
Similar approach is also used to handle Hyper-V SynIC in the
'commit 5c919412fe61 ("kvm/x86: Hyper-V synthetic interrupt controller")',
where APICv is permanently disabled at runtime (currently broken for
AVIC, and fixed by this series).
This series contains several parts:
* Part 1: patch 1,2
Code clean up, refactor, and introduce helper functions
* Part 2: patch 3
Introduce APICv deactivate bits to keep track of APICv state
for each vm.
* Part 3: patch 4-10
Add support for activate/deactivate APICv at runtime
* Part 4: patch 11-14:
Add support for various cases where APICv needs to
be deactivated
* Part 5: patch 15-17:
Introduce in-kernel IOAPIC workaround for AVIC EOI
* Part 6: path 18
Allow enable AVIC w/ kernel_irqchip=on
Pre-requisite Patch:
* commit b9c6ff94e43a ("iommu/amd: Re-factor guest virtual APIC (de-)activation code")
(https://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git/commit/
?h=next&id=b9c6ff94e43a0ee053e0c1d983fba1ac4953b762)
This series has been tested against v5.3 as following:
* Booting Linux, FreeBSD, and Windows Server 2019 VMs upto 240 vcpus
w/ qemu option "kernel-irqchip=on" and "-no-hpet".
* Pass-through Intel 10GbE NIC and run netperf in the VM.
Changes from V4: (https://lkml.org/lkml/2019/11/1/764)
* Rename APICV_DEACT_BIT_xxx to APICV_INHIBIT_REASON_xxxx
* Introduce kvm_x86_ops.check_apicv_inhibit_reasons hook
to allow vendors to specify which APICv inhibit reason bits
to support (patch 08/18).
* Update comment on kvm_request_apicv_update() no-lock requirement.
(patch 04/18)
Suravee Suthikulpanit (18):
kvm: x86: Modify kvm_x86_ops.get_enable_apicv() to use struct kvm
parameter
kvm: lapic: Introduce APICv update helper function
kvm: x86: Introduce APICv inhibit reason bits
kvm: x86: Add support for dynamic APICv
kvm: x86: Add APICv (de)activate request trace points
kvm: x86: svm: Add support to (de)activate posted interrupts
svm: Add support for setup/destroy virutal APIC backing page for AVIC
kvm: x86: Introduce APICv x86 ops for checking APIC inhibit reasons
kvm: x86: Introduce x86 ops hook for pre-update APICv
svm: Add support for dynamic APICv
kvm: x86: hyperv: Use APICv update request interface
svm: Deactivate AVIC when launching guest with nested SVM support
svm: Temporary deactivate AVIC during ExtINT handling
kvm: i8254: Deactivate APICv when using in-kernel PIT re-injection
mode.
kvm: lapic: Clean up APIC predefined macros
kvm: ioapic: Refactor kvm_ioapic_update_eoi()
kvm: ioapic: Lazy update IOAPIC EOI
svm: Allow AVIC with in-kernel irqchip mode
arch/x86/include/asm/kvm_host.h | 19 ++++-
arch/x86/kvm/hyperv.c | 5 +-
arch/x86/kvm/i8254.c | 12 +++
arch/x86/kvm/ioapic.c | 149 +++++++++++++++++++++++-------------
arch/x86/kvm/lapic.c | 35 +++++----
arch/x86/kvm/lapic.h | 2 +
arch/x86/kvm/svm.c | 164 +++++++++++++++++++++++++++++++++++-----
arch/x86/kvm/trace.h | 19 +++++
arch/x86/kvm/vmx/vmx.c | 12 ++-
arch/x86/kvm/x86.c | 71 ++++++++++++++---
10 files changed, 385 insertions(+), 103 deletions(-)
--
1.8.3.1
Add trace points when sending request to (de)activate APICv.
Suggested-by: Alexander Graf <[email protected]>
Signed-off-by: Suravee Suthikulpanit <[email protected]>
---
arch/x86/kvm/trace.h | 19 +++++++++++++++++++
arch/x86/kvm/x86.c | 2 ++
2 files changed, 21 insertions(+)
diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
index 7c741a0..f194dd0 100644
--- a/arch/x86/kvm/trace.h
+++ b/arch/x86/kvm/trace.h
@@ -1291,6 +1291,25 @@
__entry->vcpu_id, __entry->timer_index)
);
+TRACE_EVENT(kvm_apicv_update_request,
+ TP_PROTO(bool activate, unsigned long bit),
+ TP_ARGS(activate, bit),
+
+ TP_STRUCT__entry(
+ __field(bool, activate)
+ __field(unsigned long, bit)
+ ),
+
+ TP_fast_assign(
+ __entry->activate = activate;
+ __entry->bit = bit;
+ ),
+
+ TP_printk("%s bit=%lu",
+ __entry->activate ? "activate" : "deactivate",
+ __entry->bit)
+);
+
/*
* Tracepoint for AMD AVIC
*/
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 03318a2..044c628 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -7892,6 +7892,7 @@ void kvm_request_apicv_update(struct kvm *kvm, bool activate, ulong bit)
return;
}
+ trace_kvm_apicv_update_request(activate, bit);
kvm_make_all_cpus_request(kvm, KVM_REQ_APICV_UPDATE);
}
EXPORT_SYMBOL_GPL(kvm_request_apicv_update);
@@ -10282,3 +10283,4 @@ bool kvm_arch_no_poll(struct kvm_vcpu *vcpu)
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_pi_irte_update);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_avic_unaccelerated_access);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_avic_incomplete_ipi);
+EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_apicv_update_request);
--
1.8.3.1
There are several reasons in which a VM needs to deactivate APICv
e.g. disable APICv via parameter during module loading, or when
enable Hyper-V SynIC support. Additional inhibit reasons will be
introduced later on when dynamic APICv is supported,
Introduce KVM APICv inhibit reason bits along with a new variable,
apicv_inhibit_reasons, to help keep track of APICv state for each VM,
Initially, the APICV_INHIBIT_REASON_DISABLE bit is used to indicate
the case where APICv is disabled during KVM module load.
(e.g. insmod kvm_amd avic=0 or insmod kvm_intel enable_apicv=0).
Signed-off-by: Suravee Suthikulpanit <[email protected]>
---
arch/x86/include/asm/kvm_host.h | 5 +++++
arch/x86/kvm/svm.c | 13 ++++++++++++-
arch/x86/kvm/vmx/vmx.c | 1 +
arch/x86/kvm/x86.c | 20 +++++++++++++++++++-
4 files changed, 37 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 632589a..c60786a 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -847,6 +847,8 @@ enum kvm_irqchip_mode {
KVM_IRQCHIP_SPLIT, /* created with KVM_CAP_SPLIT_IRQCHIP */
};
+#define APICV_INHIBIT_REASON_DISABLE 0
+
struct kvm_arch {
unsigned long n_used_mmu_pages;
unsigned long n_requested_mmu_pages;
@@ -877,6 +879,7 @@ struct kvm_arch {
struct kvm_apic_map *apic_map;
bool apic_access_page_done;
+ unsigned long apicv_inhibit_reasons;
gpa_t wall_clock;
@@ -1441,6 +1444,8 @@ gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva,
struct x86_exception *exception);
void kvm_vcpu_deactivate_apicv(struct kvm_vcpu *vcpu);
+bool kvm_apicv_activated(struct kvm *kvm);
+void kvm_apicv_init(struct kvm *kvm, bool enable);
int kvm_emulate_hypercall(struct kvm_vcpu *vcpu);
diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c
index d53ffb8..3395e4c 100644
--- a/arch/x86/kvm/svm.c
+++ b/arch/x86/kvm/svm.c
@@ -1997,6 +1997,17 @@ static int avic_vm_init(struct kvm *kvm)
return err;
}
+static int svm_vm_init(struct kvm *kvm)
+{
+ int ret = 0;
+
+ if (avic)
+ ret = avic_vm_init(kvm);
+
+ kvm_apicv_init(kvm, (avic && !ret));
+ return ret;
+}
+
static inline int
avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu, bool r)
{
@@ -7195,7 +7206,7 @@ static bool svm_apic_init_signal_blocked(struct kvm_vcpu *vcpu)
.vm_alloc = svm_vm_alloc,
.vm_free = svm_vm_free,
- .vm_init = avic_vm_init,
+ .vm_init = svm_vm_init,
.vm_destroy = svm_vm_destroy,
.prepare_guest_switch = svm_prepare_guest_switch,
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 2aa14d5..d6d1c862 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -6848,6 +6848,7 @@ static int vmx_vm_init(struct kvm *kvm)
break;
}
}
+ kvm_apicv_init(kvm, vmx_get_enable_apicv(kvm));
return 0;
}
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 4cbb948..4d19566 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -7329,6 +7329,23 @@ void kvm_vcpu_deactivate_apicv(struct kvm_vcpu *vcpu)
kvm_x86_ops->refresh_apicv_exec_ctrl(vcpu);
}
+bool kvm_apicv_activated(struct kvm *kvm)
+{
+ return (READ_ONCE(kvm->arch.apicv_inhibit_reasons) == 0);
+}
+EXPORT_SYMBOL_GPL(kvm_apicv_activated);
+
+void kvm_apicv_init(struct kvm *kvm, bool enable)
+{
+ if (enable)
+ clear_bit(APICV_INHIBIT_REASON_DISABLE,
+ &kvm->arch.apicv_inhibit_reasons);
+ else
+ set_bit(APICV_INHIBIT_REASON_DISABLE,
+ &kvm->arch.apicv_inhibit_reasons);
+}
+EXPORT_SYMBOL_GPL(kvm_apicv_init);
+
static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id)
{
struct kvm_vcpu *target = NULL;
@@ -9347,10 +9364,11 @@ int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu)
goto fail_free_pio_data;
if (irqchip_in_kernel(vcpu->kvm)) {
- vcpu->arch.apicv_active = kvm_x86_ops->get_enable_apicv(vcpu->kvm);
r = kvm_create_lapic(vcpu, lapic_timer_advance_ns);
if (r < 0)
goto fail_mmu_destroy;
+ if (kvm_apicv_activated(vcpu->kvm))
+ vcpu->arch.apicv_active = kvm_x86_ops->get_enable_apicv(vcpu->kvm);
} else
static_key_slow_inc(&kvm_no_apic_vcpu);
--
1.8.3.1
Paolo,
Ping. Would you please let me know your feedback when you get a chance to review this series
Thanks,
Suravee
On 11/15/19 3:15 AM, Suravee Suthikulpanit wrote:
> The 'commit 67034bb9dd5e ("KVM: SVM: Add irqchip_split() checks before
> enabling AVIC")' was introduced to fix miscellaneous boot-hang issues
> when enable AVIC. This is mainly due to AVIC hardware doest not #vmexit
> on write to LAPIC EOI register resulting in-kernel PIC and IOAPIC to
> wait and do not inject new interrupts (e.g. PIT, RTC).
>
> This limits AVIC to only work with kernel_irqchip=split mode, which is
> not currently enabled by default, and also required user-space to
> support split irqchip model, which might not be the case.
>
> The goal of this series is to enable AVIC to work in both irqchip modes,
> by allowing AVIC to be deactivated temporarily during runtime, and fallback
> to legacy interrupt injection mode (w/ vINTR and interrupt windows)
> when needed, and then re-enabled subsequently (a.k.a Dynamic APICv).
>
> Similar approach is also used to handle Hyper-V SynIC in the
> 'commit 5c919412fe61 ("kvm/x86: Hyper-V synthetic interrupt controller")',
> where APICv is permanently disabled at runtime (currently broken for
> AVIC, and fixed by this series).
>
> This series contains several parts:
> * Part 1: patch 1,2
> Code clean up, refactor, and introduce helper functions
>
> * Part 2: patch 3
> Introduce APICv deactivate bits to keep track of APICv state
> for each vm.
>
> * Part 3: patch 4-10
> Add support for activate/deactivate APICv at runtime
>
> * Part 4: patch 11-14:
> Add support for various cases where APICv needs to
> be deactivated
>
> * Part 5: patch 15-17:
> Introduce in-kernel IOAPIC workaround for AVIC EOI
>
> * Part 6: path 18
> Allow enable AVIC w/ kernel_irqchip=on
>
> Pre-requisite Patch:
> * commit b9c6ff94e43a ("iommu/amd: Re-factor guest virtual APIC (de-)activation code")
> (https://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git/commit/
> ?h=next&id=b9c6ff94e43a0ee053e0c1d983fba1ac4953b762)
>
> This series has been tested against v5.3 as following:
> * Booting Linux, FreeBSD, and Windows Server 2019 VMs upto 240 vcpus
> w/ qemu option "kernel-irqchip=on" and "-no-hpet".
> * Pass-through Intel 10GbE NIC and run netperf in the VM.
>
> Changes from V4: (https://lkml.org/lkml/2019/11/1/764)
> * Rename APICV_DEACT_BIT_xxx to APICV_INHIBIT_REASON_xxxx
> * Introduce kvm_x86_ops.check_apicv_inhibit_reasons hook
> to allow vendors to specify which APICv inhibit reason bits
> to support (patch 08/18).
> * Update comment on kvm_request_apicv_update() no-lock requirement.
> (patch 04/18)
>
> Suravee Suthikulpanit (18):
> kvm: x86: Modify kvm_x86_ops.get_enable_apicv() to use struct kvm
> parameter
> kvm: lapic: Introduce APICv update helper function
> kvm: x86: Introduce APICv inhibit reason bits
> kvm: x86: Add support for dynamic APICv
> kvm: x86: Add APICv (de)activate request trace points
> kvm: x86: svm: Add support to (de)activate posted interrupts
> svm: Add support for setup/destroy virutal APIC backing page for AVIC
> kvm: x86: Introduce APICv x86 ops for checking APIC inhibit reasons
> kvm: x86: Introduce x86 ops hook for pre-update APICv
> svm: Add support for dynamic APICv
> kvm: x86: hyperv: Use APICv update request interface
> svm: Deactivate AVIC when launching guest with nested SVM support
> svm: Temporary deactivate AVIC during ExtINT handling
> kvm: i8254: Deactivate APICv when using in-kernel PIT re-injection
> mode.
> kvm: lapic: Clean up APIC predefined macros
> kvm: ioapic: Refactor kvm_ioapic_update_eoi()
> kvm: ioapic: Lazy update IOAPIC EOI
> svm: Allow AVIC with in-kernel irqchip mode
>
> arch/x86/include/asm/kvm_host.h | 19 ++++-
> arch/x86/kvm/hyperv.c | 5 +-
> arch/x86/kvm/i8254.c | 12 +++
> arch/x86/kvm/ioapic.c | 149 +++++++++++++++++++++++-------------
> arch/x86/kvm/lapic.c | 35 +++++----
> arch/x86/kvm/lapic.h | 2 +
> arch/x86/kvm/svm.c | 164 +++++++++++++++++++++++++++++++++++-----
> arch/x86/kvm/trace.h | 19 +++++
> arch/x86/kvm/vmx/vmx.c | 12 ++-
> arch/x86/kvm/x86.c | 71 ++++++++++++++---
> 10 files changed, 385 insertions(+), 103 deletions(-)
>
Ping
Thanks
Suravee
On 1/2/20 5:17 PM, Suravee Suthikulpanit wrote:
> Paolo,
>
> Ping. Would you please let me know your feedback when you get a chance to review this series
>
> Thanks,
> Suravee
>
> On 11/15/19 3:15 AM, Suravee Suthikulpanit wrote:
>> The 'commit 67034bb9dd5e ("KVM: SVM: Add irqchip_split() checks before
>> enabling AVIC")' was introduced to fix miscellaneous boot-hang issues
>> when enable AVIC. This is mainly due to AVIC hardware doest not #vmexit
>> on write to LAPIC EOI register resulting in-kernel PIC and IOAPIC to
>> wait and do not inject new interrupts (e.g. PIT, RTC).
>>
>> This limits AVIC to only work with kernel_irqchip=split mode, which is
>> not currently enabled by default, and also required user-space to
>> support split irqchip model, which might not be the case.
>>
>> The goal of this series is to enable AVIC to work in both irqchip modes,
>> by allowing AVIC to be deactivated temporarily during runtime, and fallback
>> to legacy interrupt injection mode (w/ vINTR and interrupt windows)
>> when needed, and then re-enabled subsequently (a.k.a Dynamic APICv).
>>
>> Similar approach is also used to handle Hyper-V SynIC in the
>> 'commit 5c919412fe61 ("kvm/x86: Hyper-V synthetic interrupt controller")',
>> where APICv is permanently disabled at runtime (currently broken for
>> AVIC, and fixed by this series).
>>
>> This series contains several parts:
>> * Part 1: patch 1,2
>> Code clean up, refactor, and introduce helper functions
>>
>> * Part 2: patch 3
>> Introduce APICv deactivate bits to keep track of APICv state
>> for each vm.
>> * Part 3: patch 4-10
>> Add support for activate/deactivate APICv at runtime
>>
>> * Part 4: patch 11-14:
>> Add support for various cases where APICv needs to
>> be deactivated
>>
>> * Part 5: patch 15-17:
>> Introduce in-kernel IOAPIC workaround for AVIC EOI
>>
>> * Part 6: path 18
>> Allow enable AVIC w/ kernel_irqchip=on
>>
>> Pre-requisite Patch:
>> * commit b9c6ff94e43a ("iommu/amd: Re-factor guest virtual APIC (de-)activation code")
>> (https://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git/commit/
>> ?h=next&id=b9c6ff94e43a0ee053e0c1d983fba1ac4953b762)
>>
>> This series has been tested against v5.3 as following:
>> * Booting Linux, FreeBSD, and Windows Server 2019 VMs upto 240 vcpus
>> w/ qemu option "kernel-irqchip=on" and "-no-hpet".
>> * Pass-through Intel 10GbE NIC and run netperf in the VM.
>>
>> Changes from V4: (https://lkml.org/lkml/2019/11/1/764)
>> * Rename APICV_DEACT_BIT_xxx to APICV_INHIBIT_REASON_xxxx
>> * Introduce kvm_x86_ops.check_apicv_inhibit_reasons hook
>> to allow vendors to specify which APICv inhibit reason bits
>> to support (patch 08/18).
>> * Update comment on kvm_request_apicv_update() no-lock requirement.
>> (patch 04/18)
>>
>> Suravee Suthikulpanit (18):
>> kvm: x86: Modify kvm_x86_ops.get_enable_apicv() to use struct kvm
>> parameter
>> kvm: lapic: Introduce APICv update helper function
>> kvm: x86: Introduce APICv inhibit reason bits
>> kvm: x86: Add support for dynamic APICv
>> kvm: x86: Add APICv (de)activate request trace points
>> kvm: x86: svm: Add support to (de)activate posted interrupts
>> svm: Add support for setup/destroy virutal APIC backing page for AVIC
>> kvm: x86: Introduce APICv x86 ops for checking APIC inhibit reasons
>> kvm: x86: Introduce x86 ops hook for pre-update APICv
>> svm: Add support for dynamic APICv
>> kvm: x86: hyperv: Use APICv update request interface
>> svm: Deactivate AVIC when launching guest with nested SVM support
>> svm: Temporary deactivate AVIC during ExtINT handling
>> kvm: i8254: Deactivate APICv when using in-kernel PIT re-injection
>> mode.
>> kvm: lapic: Clean up APIC predefined macros
>> kvm: ioapic: Refactor kvm_ioapic_update_eoi()
>> kvm: ioapic: Lazy update IOAPIC EOI
>> svm: Allow AVIC with in-kernel irqchip mode
>>
>> arch/x86/include/asm/kvm_host.h | 19 ++++-
>> arch/x86/kvm/hyperv.c | 5 +-
>> arch/x86/kvm/i8254.c | 12 +++
>> arch/x86/kvm/ioapic.c | 149 +++++++++++++++++++++++-------------
>> arch/x86/kvm/lapic.c | 35 +++++----
>> arch/x86/kvm/lapic.h | 2 +
>> arch/x86/kvm/svm.c | 164 +++++++++++++++++++++++++++++++++++-----
>> arch/x86/kvm/trace.h | 19 +++++
>> arch/x86/kvm/vmx/vmx.c | 12 ++-
>> arch/x86/kvm/x86.c | 71 ++++++++++++++---
>> 10 files changed, 385 insertions(+), 103 deletions(-)
>>
A couple slight improvements, after which get_enable_apicv is unused.
Paolo
On 14/11/19 21:15, Suravee Suthikulpanit wrote:
> @@ -6848,6 +6848,7 @@ static int vmx_vm_init(struct kvm *kvm)
> break;
> }
> }
> + kvm_apicv_init(kvm, vmx_get_enable_apicv(kvm));
> return 0;
> }
>
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 01f60d6cd881..d011ffaec7ee 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -6852,7 +6852,7 @@ static int vmx_vm_init(struct kvm *kvm)
break;
}
}
- kvm_apicv_init(kvm, vmx_get_enable_apicv(kvm));
+ kvm_apicv_init(kvm, enable_apicv);
return 0;
}
> @@ -9347,10 +9364,11 @@ int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu)
> goto fail_free_pio_data;
>
> if (irqchip_in_kernel(vcpu->kvm)) {
> - vcpu->arch.apicv_active = kvm_x86_ops->get_enable_apicv(vcpu->kvm);
> r = kvm_create_lapic(vcpu, lapic_timer_advance_ns);
> if (r < 0)
> goto fail_mmu_destroy;
> + if (kvm_apicv_activated(vcpu->kvm))
> + vcpu->arch.apicv_active = kvm_x86_ops->get_enable_apicv(vcpu->kvm);
> } else
> static_key_slow_inc(&kvm_no_apic_vcpu);
>
>
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 6c62abb4fdd9..cc9f84527a55 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -9236,7 +9236,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
if (r < 0)
goto fail_mmu_destroy;
if (kvm_apicv_activated(vcpu->kvm))
- vcpu->arch.apicv_active = kvm_x86_ops->get_enable_apicv(vcpu->kvm);
+ vcpu->arch.apicv_active = true;
} else
static_key_slow_inc(&kvm_no_apic_vcpu);
On 20/01/20 07:16, Suravee Suthikulpanit wrote:
> Ping
>
> Thanks
> Suravee
Queued it, finally. Sorry for the wait.
Paolo
> On 1/2/20 5:17 PM, Suravee Suthikulpanit wrote:
>> Paolo,
>>
>> Ping. Would you please let me know your feedback when you get a chance
>> to review this series
>>
>> Thanks,
>> Suravee
>>
>> On 11/15/19 3:15 AM, Suravee Suthikulpanit wrote:
>>> The 'commit 67034bb9dd5e ("KVM: SVM: Add irqchip_split() checks before
>>> enabling AVIC")' was introduced to fix miscellaneous boot-hang issues
>>> when enable AVIC. This is mainly due to AVIC hardware doest not #vmexit
>>> on write to LAPIC EOI register resulting in-kernel PIC and IOAPIC to
>>> wait and do not inject new interrupts (e.g. PIT, RTC).
>>>
>>> This limits AVIC to only work with kernel_irqchip=split mode, which is
>>> not currently enabled by default, and also required user-space to
>>> support split irqchip model, which might not be the case.
>>>
>>> The goal of this series is to enable AVIC to work in both irqchip modes,
>>> by allowing AVIC to be deactivated temporarily during runtime, and
>>> fallback
>>> to legacy interrupt injection mode (w/ vINTR and interrupt windows)
>>> when needed, and then re-enabled subsequently (a.k.a Dynamic APICv).
>>>
>>> Similar approach is also used to handle Hyper-V SynIC in the
>>> 'commit 5c919412fe61 ("kvm/x86: Hyper-V synthetic interrupt
>>> controller")',
>>> where APICv is permanently disabled at runtime (currently broken for
>>> AVIC, and fixed by this series).
>>>
>>> This series contains several parts:
>>> * Part 1: patch 1,2
>>> Code clean up, refactor, and introduce helper functions
>>>
>>> * Part 2: patch 3
>>> Introduce APICv deactivate bits to keep track of APICv state
>>> for each vm.
>>> * Part 3: patch 4-10
>>> Add support for activate/deactivate APICv at runtime
>>>
>>> * Part 4: patch 11-14:
>>> Add support for various cases where APICv needs to
>>> be deactivated
>>>
>>> * Part 5: patch 15-17:
>>> Introduce in-kernel IOAPIC workaround for AVIC EOI
>>>
>>> * Part 6: path 18
>>> Allow enable AVIC w/ kernel_irqchip=on
>>>
>>> Pre-requisite Patch:
>>> * commit b9c6ff94e43a ("iommu/amd: Re-factor guest virtual APIC
>>> (de-)activation code")
>>>
>>> (https://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git/commit/
>>> ?h=next&id=b9c6ff94e43a0ee053e0c1d983fba1ac4953b762)
>>>
>>> This series has been tested against v5.3 as following:
>>> * Booting Linux, FreeBSD, and Windows Server 2019 VMs upto 240 vcpus
>>> w/ qemu option "kernel-irqchip=on" and "-no-hpet".
>>> * Pass-through Intel 10GbE NIC and run netperf in the VM.
>>>
>>> Changes from V4: (https://lkml.org/lkml/2019/11/1/764)
>>> * Rename APICV_DEACT_BIT_xxx to APICV_INHIBIT_REASON_xxxx
>>> * Introduce kvm_x86_ops.check_apicv_inhibit_reasons hook
>>> to allow vendors to specify which APICv inhibit reason bits
>>> to support (patch 08/18).
>>> * Update comment on kvm_request_apicv_update() no-lock requirement.
>>> (patch 04/18)
>>>
>>> Suravee Suthikulpanit (18):
>>> kvm: x86: Modify kvm_x86_ops.get_enable_apicv() to use struct kvm
>>> parameter
>>> kvm: lapic: Introduce APICv update helper function
>>> kvm: x86: Introduce APICv inhibit reason bits
>>> kvm: x86: Add support for dynamic APICv
>>> kvm: x86: Add APICv (de)activate request trace points
>>> kvm: x86: svm: Add support to (de)activate posted interrupts
>>> svm: Add support for setup/destroy virutal APIC backing page for AVIC
>>> kvm: x86: Introduce APICv x86 ops for checking APIC inhibit reasons
>>> kvm: x86: Introduce x86 ops hook for pre-update APICv
>>> svm: Add support for dynamic APICv
>>> kvm: x86: hyperv: Use APICv update request interface
>>> svm: Deactivate AVIC when launching guest with nested SVM support
>>> svm: Temporary deactivate AVIC during ExtINT handling
>>> kvm: i8254: Deactivate APICv when using in-kernel PIT re-injection
>>> mode.
>>> kvm: lapic: Clean up APIC predefined macros
>>> kvm: ioapic: Refactor kvm_ioapic_update_eoi()
>>> kvm: ioapic: Lazy update IOAPIC EOI
>>> svm: Allow AVIC with in-kernel irqchip mode
>>>
>>> arch/x86/include/asm/kvm_host.h | 19 ++++-
>>> arch/x86/kvm/hyperv.c | 5 +-
>>> arch/x86/kvm/i8254.c | 12 +++
>>> arch/x86/kvm/ioapic.c | 149
>>> +++++++++++++++++++++++-------------
>>> arch/x86/kvm/lapic.c | 35 +++++----
>>> arch/x86/kvm/lapic.h | 2 +
>>> arch/x86/kvm/svm.c | 164
>>> +++++++++++++++++++++++++++++++++++-----
>>> arch/x86/kvm/trace.h | 19 +++++
>>> arch/x86/kvm/vmx/vmx.c | 12 ++-
>>> arch/x86/kvm/x86.c | 71 ++++++++++++++---
>>> 10 files changed, 385 insertions(+), 103 deletions(-)
>>>
>