2014-12-21 12:53:26

by Jamie Heilman

Subject: regression bisected; KVM: entry failed, hardware error 0x80000021

With v3.19-rc1 when I run qemu-system-x86_64 -machine pc,accel=kvm I
get:

KVM: entry failed, hardware error 0x80000021

If you're running a guest on an Intel machine without unrestricted mode
support, the failure can be most likely due to the guest entering an invalid
state for Intel VT. For example, the guest maybe running in big real mode
which is not supported on less recent Intel processors.

EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000e05b EFL=00010002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 000f0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 00000000 0000ffff
IDT= 00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=85 00 87 00 89 00 8b 00 00 00 86 00 88 00 8a 00 8c 00 00 90 <2e> 66 83 3e 30 6c 00 0f 85 e7 f2 31 c0 8e d0 66 bc 00 70 00 00 66 ba 31 2e 0f 00 e9 45 f1

This is with QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-11),
Copyright (c) 2003-2008 Fabrice Bellard

The host system is:

cpu family : 6
model : 23
model name : Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
stepping : 10
microcode : 0xa0b
...
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm dtherm tpr_shadow vnmi flexpriority

I bisected this back to:

commit 34a1cd60d17f62c1f077c1478a6c2ca8c3d17af4
Author: Tiejun Chen <[email protected]>
Date: Tue Oct 28 10:14:48 2014 +0800

kvm: x86: vmx: move some vmx setting from vmx_init() to hardware_setup()

Instead of vmx_init(), actually it would make reasonable sense to do
anything specific to vmx hardware setting in vmx_x86_ops->hardware_setup().

Signed-off-by: Tiejun Chen <[email protected]>
Signed-off-by: Paolo Bonzini <[email protected]>

reverting this commit appears to fix the issue, in so far that I can
run kvm again without the aforementioned error, but I haven't tested
it thoroughly beyond that. Let me know if you need more information
or testing.

--
Jamie Heilman http://audible.transient.net/~jamie/

2014-12-22 07:30:21

by Chen, Tiejun

Subject: Re: regression bisected; KVM: entry failed, hardware error 0x80000021

On 2014/12/21 20:46, Jamie Heilman wrote:
> With v3.19-rc1 when I run qemu-system-x86_64 -machine pc,accel=kvm I
> get:
>
> KVM: entry failed, hardware error 0x80000021

Looks some MSR writing issues such a failed entry.

>
> If you're running a guest on an Intel machine without unrestricted mode
> support, the failure can be most likely due to the guest entering an invalid
> state for Intel VT. For example, the guest maybe running in big real mode
> which is not supported on less recent Intel processors.
>
> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
> EIP=0000e05b EFL=00010002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0000 00000000 0000ffff 00009300
> CS =f000 000f0000 0000ffff 00009b00
> SS =0000 00000000 0000ffff 00009300
> DS =0000 00000000 0000ffff 00009300
> FS =0000 00000000 0000ffff 00009300
> GS =0000 00000000 0000ffff 00009300
> LDT=0000 00000000 0000ffff 00008200
> TR =0000 00000000 0000ffff 00008b00
> GDT= 00000000 0000ffff
> IDT= 00000000 0000ffff
> CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000

And I don't see any obvious wrong as well. Any valuable info from dmesg?

> Code=85 00 87 00 89 00 8b 00 00 00 86 00 88 00 8a 00 8c 00 00 90 <2e> 66 83 3e 30 6c 00 0f 85 e7 f2 31 c0 8e d0 66 bc 00 70 00 00 66 ba 31 2e 0f 00 e9 45 f1
>
> This is with QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-11),
> Copyright (c) 2003-2008 Fabrice Bellard
>
> The host system is:
>
> cpu family : 6
> model : 23
> model name : Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
> stepping : 10
> microcode : 0xa0b
> ...
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm dtherm tpr_shadow vnmi flexpriority
>
> I bisected this back to:
>
> commit 34a1cd60d17f62c1f077c1478a6c2ca8c3d17af4
> Author: Tiejun Chen <[email protected]>
> Date: Tue Oct 28 10:14:48 2014 +0800
>
> kvm: x86: vmx: move some vmx setting from vmx_init() to hardware_setup()
>
> Instead of vmx_init(), actually it would make reasonable sense to do
> anything specific to vmx hardware setting in vmx_x86_ops->hardware_setup().
>

This commit just reorders something but some MSR writing depend on
previous status.

Could you try this?

diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index feb852b..96c84a8 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -5840,49 +5840,6 @@ static __init int hardware_setup(void)
memset(vmx_msr_bitmap_legacy, 0xff, PAGE_SIZE);
memset(vmx_msr_bitmap_longmode, 0xff, PAGE_SIZE);

- vmx_disable_intercept_for_msr(MSR_FS_BASE, false);
- vmx_disable_intercept_for_msr(MSR_GS_BASE, false);
- vmx_disable_intercept_for_msr(MSR_KERNEL_GS_BASE, true);
- vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_CS, false);
- vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_ESP, false);
- vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_EIP, false);
- vmx_disable_intercept_for_msr(MSR_IA32_BNDCFGS, true);
-
- memcpy(vmx_msr_bitmap_legacy_x2apic,
- vmx_msr_bitmap_legacy, PAGE_SIZE);
- memcpy(vmx_msr_bitmap_longmode_x2apic,
- vmx_msr_bitmap_longmode, PAGE_SIZE);
-
- if (enable_apicv) {
- for (msr = 0x800; msr <= 0x8ff; msr++)
- vmx_disable_intercept_msr_read_x2apic(msr);
-
- /* According SDM, in x2apic mode, the whole id reg is used.
- * But in KVM, it only use the highest eight bits. Need to
- * intercept it */
- vmx_enable_intercept_msr_read_x2apic(0x802);
- /* TMCCT */
- vmx_enable_intercept_msr_read_x2apic(0x839);
- /* TPR */
- vmx_disable_intercept_msr_write_x2apic(0x808);
- /* EOI */
- vmx_disable_intercept_msr_write_x2apic(0x80b);
- /* SELF-IPI */
- vmx_disable_intercept_msr_write_x2apic(0x83f);
- }
-
- if (enable_ept) {
- kvm_mmu_set_mask_ptes(0ull,
- (enable_ept_ad_bits) ? VMX_EPT_ACCESS_BIT : 0ull,
- (enable_ept_ad_bits) ? VMX_EPT_DIRTY_BIT : 0ull,
- 0ull, VMX_EPT_EXECUTABLE_MASK);
- ept_set_mmio_spte_mask();
- kvm_enable_tdp();
- } else
- kvm_disable_tdp();
-
- update_ple_window_actual_max();
-
if (setup_vmcs_config(&vmcs_config) < 0) {
r = -EIO;
goto out7;
@@ -5945,6 +5902,49 @@ static __init int hardware_setup(void)
if (nested)
nested_vmx_setup_ctls_msrs();

+ vmx_disable_intercept_for_msr(MSR_FS_BASE, false);
+ vmx_disable_intercept_for_msr(MSR_GS_BASE, false);
+ vmx_disable_intercept_for_msr(MSR_KERNEL_GS_BASE, true);
+ vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_CS, false);
+ vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_ESP, false);
+ vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_EIP, false);
+ vmx_disable_intercept_for_msr(MSR_IA32_BNDCFGS, true);
+
+ memcpy(vmx_msr_bitmap_legacy_x2apic,
+ vmx_msr_bitmap_legacy, PAGE_SIZE);
+ memcpy(vmx_msr_bitmap_longmode_x2apic,
+ vmx_msr_bitmap_longmode, PAGE_SIZE);
+
+ if (enable_apicv) {
+ for (msr = 0x800; msr <= 0x8ff; msr++)
+ vmx_disable_intercept_msr_read_x2apic(msr);
+
+ /* According SDM, in x2apic mode, the whole id reg is used.
+ * But in KVM, it only use the highest eight bits. Need to
+ * intercept it */
+ vmx_enable_intercept_msr_read_x2apic(0x802);
+ /* TMCCT */
+ vmx_enable_intercept_msr_read_x2apic(0x839);
+ /* TPR */
+ vmx_disable_intercept_msr_write_x2apic(0x808);
+ /* EOI */
+ vmx_disable_intercept_msr_write_x2apic(0x80b);
+ /* SELF-IPI */
+ vmx_disable_intercept_msr_write_x2apic(0x83f);
+ }
+
+ if (enable_ept) {
+ kvm_mmu_set_mask_ptes(0ull,
+ (enable_ept_ad_bits) ? VMX_EPT_ACCESS_BIT : 0ull,
+ (enable_ept_ad_bits) ? VMX_EPT_DIRTY_BIT : 0ull,
+ 0ull, VMX_EPT_EXECUTABLE_MASK);
+ ept_set_mmio_spte_mask();
+ kvm_enable_tdp();
+ } else
+ kvm_disable_tdp();
+
+ update_ple_window_actual_max();
+
return alloc_kvm_area();

out7:


Thanks
Tiejun

2014-12-22 09:24:05

by Jamie Heilman

Subject: Re: regression bisected; KVM: entry failed, hardware error 0x80000021

Chen, Tiejun wrote:
> On 2014/12/21 20:46, Jamie Heilman wrote:
> >With v3.19-rc1 when I run qemu-system-x86_64 -machine pc,accel=kvm I
> >get:
> >
> >KVM: entry failed, hardware error 0x80000021
>
> Looks some MSR writing issues such a failed entry.
>
> >If you're running a guest on an Intel machine without unrestricted mode
> >support, the failure can be most likely due to the guest entering an invalid
> >state for Intel VT. For example, the guest maybe running in big real mode
> >which is not supported on less recent Intel processors.
> >
> >EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
> >ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
> >EIP=0000e05b EFL=00010002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> >ES =0000 00000000 0000ffff 00009300
> >CS =f000 000f0000 0000ffff 00009b00
> >SS =0000 00000000 0000ffff 00009300
> >DS =0000 00000000 0000ffff 00009300
> >FS =0000 00000000 0000ffff 00009300
> >GS =0000 00000000 0000ffff 00009300
> >LDT=0000 00000000 0000ffff 00008200
> >TR =0000 00000000 0000ffff 00008b00
> >GDT= 00000000 0000ffff
> >IDT= 00000000 0000ffff
> >CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
> >DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
> >DR6=00000000ffff0ff0 DR7=0000000000000400
> >EFER=0000000000000000
>
> And I don't see any obvious wrong as well. Any valuable info from dmesg?

With the simple qemu command above, on 3.18.1 I see:

kern.info: kvm: zapping shadow pages for mmio generation wraparound

when I fire up a full guest that's actually useful I get:

kern.info: kvm: zapping shadow pages for mmio generation wraparound
kern.err: kvm [4073]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff

On 3.18.0-rc3-00042-g34a1cd6 nothing appears in the dmesg, just the
message I mention above to stderr. Same thing with a stock
3.19.0-rc1. Once I apply your patch the simple test command produces
the same zapping shadow pages messages as 3.18.1, and a test guest of
a Debian Jessie image (w/stock distro kernel) produces the same thing
with disabled perfctr wrmsr message. However, it doesn't look like
I'm entirely out of the woods, because one of my other guest VMs with a
custom kernel that works great under 3.18.1 now fails to run. Nothing
in dmesg, but here's the stderr:

KVM internal error. Suberror: 1
emulation failure
EAX=000de494 EBX=00000000 ECX=00000000 EDX=00000cfd
ESI=00000059 EDI=00000000 EBP=00000000 ESP=00006fb4
EIP=000f15c1 EFL=00010016 [----AP-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT= 000f6be8 00000037
IDT= 000f6c26 00000000
CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=e8 ae fc ff ff 89 f2 a8 10 89 d8 75 0a b9 41 15 ff ff ff d1 <5b> 5e c3 5b 5e e9 76 ff ff ff b0 11 e6 20 e6 a0 b0 08 e6 21 b0 70 e6 a1 b0 04 e6 21 b0 02

FWIW, I get the same thing with 34a1cd60d17 reverted. Maybe there are
two bugs, maybe there's more to this first one. I can repro this
error with the command: qemu-system-x86_64 -machine pc,accel=kvm -nodefaults

> >This is with QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-11),
> >Copyright (c) 2003-2008 Fabrice Bellard
> >
> >The host system is:
> >
> >cpu family : 6
> >model : 23
> >model name : Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
> >stepping : 10
> >microcode : 0xa0b
> >...
> >flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm dtherm tpr_shadow vnmi flexpriority
> >
> >I bisected this back to:
> >
> >commit 34a1cd60d17f62c1f077c1478a6c2ca8c3d17af4
> >Author: Tiejun Chen <[email protected]>
> >Date: Tue Oct 28 10:14:48 2014 +0800
> >
> > kvm: x86: vmx: move some vmx setting from vmx_init() to hardware_setup()
> >
> > Instead of vmx_init(), actually it would make reasonable sense to do
> > anything specific to vmx hardware setting in vmx_x86_ops->hardware_setup().
> >
>
> This commit just reorders something but some MSR writing depend on previous
> status.
>
> Could you try this?

I unmangled the expanded tabs, and applied this:

diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index feb852b..96c84a8 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -5840,49 +5840,6 @@ static __init int hardware_setup(void)
memset(vmx_msr_bitmap_legacy, 0xff, PAGE_SIZE);
memset(vmx_msr_bitmap_longmode, 0xff, PAGE_SIZE);

- vmx_disable_intercept_for_msr(MSR_FS_BASE, false);
- vmx_disable_intercept_for_msr(MSR_GS_BASE, false);
- vmx_disable_intercept_for_msr(MSR_KERNEL_GS_BASE, true);
- vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_CS, false);
- vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_ESP, false);
- vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_EIP, false);
- vmx_disable_intercept_for_msr(MSR_IA32_BNDCFGS, true);
-
- memcpy(vmx_msr_bitmap_legacy_x2apic,
- vmx_msr_bitmap_legacy, PAGE_SIZE);
- memcpy(vmx_msr_bitmap_longmode_x2apic,
- vmx_msr_bitmap_longmode, PAGE_SIZE);
-
- if (enable_apicv) {
- for (msr = 0x800; msr <= 0x8ff; msr++)
- vmx_disable_intercept_msr_read_x2apic(msr);
-
- /* According SDM, in x2apic mode, the whole id reg is used.
- * But in KVM, it only use the highest eight bits. Need to
- * intercept it */
- vmx_enable_intercept_msr_read_x2apic(0x802);
- /* TMCCT */
- vmx_enable_intercept_msr_read_x2apic(0x839);
- /* TPR */
- vmx_disable_intercept_msr_write_x2apic(0x808);
- /* EOI */
- vmx_disable_intercept_msr_write_x2apic(0x80b);
- /* SELF-IPI */
- vmx_disable_intercept_msr_write_x2apic(0x83f);
- }
-
- if (enable_ept) {
- kvm_mmu_set_mask_ptes(0ull,
- (enable_ept_ad_bits) ? VMX_EPT_ACCESS_BIT : 0ull,
- (enable_ept_ad_bits) ? VMX_EPT_DIRTY_BIT : 0ull,
- 0ull, VMX_EPT_EXECUTABLE_MASK);
- ept_set_mmio_spte_mask();
- kvm_enable_tdp();
- } else
- kvm_disable_tdp();
-
- update_ple_window_actual_max();
-
if (setup_vmcs_config(&vmcs_config) < 0) {
r = -EIO;
goto out7;
@@ -5945,6 +5902,49 @@ static __init int hardware_setup(void)
if (nested)
nested_vmx_setup_ctls_msrs();

+ vmx_disable_intercept_for_msr(MSR_FS_BASE, false);
+ vmx_disable_intercept_for_msr(MSR_GS_BASE, false);
+ vmx_disable_intercept_for_msr(MSR_KERNEL_GS_BASE, true);
+ vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_CS, false);
+ vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_ESP, false);
+ vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_EIP, false);
+ vmx_disable_intercept_for_msr(MSR_IA32_BNDCFGS, true);
+
+ memcpy(vmx_msr_bitmap_legacy_x2apic,
+ vmx_msr_bitmap_legacy, PAGE_SIZE);
+ memcpy(vmx_msr_bitmap_longmode_x2apic,
+ vmx_msr_bitmap_longmode, PAGE_SIZE);
+
+ if (enable_apicv) {
+ for (msr = 0x800; msr <= 0x8ff; msr++)
+ vmx_disable_intercept_msr_read_x2apic(msr);
+
+ /* According SDM, in x2apic mode, the whole id reg is used.
+ * But in KVM, it only use the highest eight bits. Need to
+ * intercept it */
+ vmx_enable_intercept_msr_read_x2apic(0x802);
+ /* TMCCT */
+ vmx_enable_intercept_msr_read_x2apic(0x839);
+ /* TPR */
+ vmx_disable_intercept_msr_write_x2apic(0x808);
+ /* EOI */
+ vmx_disable_intercept_msr_write_x2apic(0x80b);
+ /* SELF-IPI */
+ vmx_disable_intercept_msr_write_x2apic(0x83f);
+ }
+
+ if (enable_ept) {
+ kvm_mmu_set_mask_ptes(0ull,
+ (enable_ept_ad_bits) ? VMX_EPT_ACCESS_BIT : 0ull,
+ (enable_ept_ad_bits) ? VMX_EPT_DIRTY_BIT : 0ull,
+ 0ull, VMX_EPT_EXECUTABLE_MASK);
+ ept_set_mmio_spte_mask();
+ kvm_enable_tdp();
+ } else
+ kvm_disable_tdp();
+
+ update_ple_window_actual_max();
+
return alloc_kvm_area();

out7:


--
Jamie Heilman http://audible.transient.net/~jamie/

2014-12-23 01:50:22

by Chen, Tiejun

Subject: Re: regression bisected; KVM: entry failed, hardware error 0x80000021

On 2014/12/22 17:23, Jamie Heilman wrote:
> Chen, Tiejun wrote:
>> On 2014/12/21 20:46, Jamie Heilman wrote:
>>> With v3.19-rc1 when I run qemu-system-x86_64 -machine pc,accel=kvm I
>>> get:
>>>
>>> KVM: entry failed, hardware error 0x80000021
>>
>> Looks some MSR writing issues such a failed entry.
>>
>>> If you're running a guest on an Intel machine without unrestricted mode
>>> support, the failure can be most likely due to the guest entering an invalid
>>> state for Intel VT. For example, the guest maybe running in big real mode
>>> which is not supported on less recent Intel processors.
>>>
>>> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
>>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
>>> EIP=0000e05b EFL=00010002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>> ES =0000 00000000 0000ffff 00009300
>>> CS =f000 000f0000 0000ffff 00009b00
>>> SS =0000 00000000 0000ffff 00009300
>>> DS =0000 00000000 0000ffff 00009300
>>> FS =0000 00000000 0000ffff 00009300
>>> GS =0000 00000000 0000ffff 00009300
>>> LDT=0000 00000000 0000ffff 00008200
>>> TR =0000 00000000 0000ffff 00008b00
>>> GDT= 00000000 0000ffff
>>> IDT= 00000000 0000ffff
>>> CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>>> EFER=0000000000000000
>>
>> And I don't see any obvious wrong as well. Any valuable info from dmesg?
>
> With the simple qemu command above, on 3.18.1 I see:
>
> kern.info: kvm: zapping shadow pages for mmio generation wraparound
>
> when I fire up a full guest that's actually useful I get:
>
> kern.info: kvm: zapping shadow pages for mmio generation wraparound
> kern.err: kvm [4073]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff
>
> On 3.18.0-rc3-00042-g34a1cd6 nothing appears in the dmesg, just the
> message I mention above to stderr. Same thing with a stock
> 3.19.0-rc1. Once I apply your patch the simple test command produces
> the same zapping shadow pages messages as 3.18.1, and a test guest of
> a Debian Jessie image (w/stock distro kernel) produces the same thing
> with disabled perfctr wrmsr message. However, it doesn't look like

Sorry I'm not sure if I understood current status. Looks 3.19-rc1 & my
patch just fix that error above,

KVM: entry failed, hardware error 0x80000021
...

Right?

> I'm entirely out of the woods, because one of my other guest VMs with a
> custom kernel that works great under 3.18.1 now fails to run. Nothing
> in dmesg, but here's the stderr:

But even you revert 34a1cd60d17 or just apply my patch, something else
introduced between 3.18.1 and 3.19-rc1 led this error below, right?

>
> KVM internal error. Suberror: 1
> emulation failure
> EAX=000de494 EBX=00000000 ECX=00000000 EDX=00000cfd
> ESI=00000059 EDI=00000000 EBP=00000000 ESP=00006fb4
> EIP=000f15c1 EFL=00010016 [----AP-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
> TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
> GDT= 000f6be8 00000037
> IDT= 000f6c26 00000000
> CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=e8 ae fc ff ff 89 f2 a8 10 89 d8 75 0a b9 41 15 ff ff ff d1 <5b> 5e c3 5b 5e e9 76 ff ff ff b0 11 e6 20 e6 a0 b0 08 e6 21 b0 70 e6 a1 b0 04 e6 21 b0 02
>
> FWIW, I get the same thing with 34a1cd60d17 reverted. Maybe there are
> two bugs, maybe there's more to this first one. I can repro this

So if my understanding is correct, this is probably another bug. And
especially, I already saw the same log in another thread, "Cleaning up
the KVM clock". Maybe you can continue to `git bisect` to locate that
bad commit.

Thanks
Tiejun

> error with the command: qemu-system-x86_64 -machine pc,accel=kvm -nodefaults
>
>>> This is with QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-11),
>>> Copyright (c) 2003-2008 Fabrice Bellard
>>>
>>> The host system is:
>>>
>>> cpu family : 6
>>> model : 23
>>> model name : Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
>>> stepping : 10
>>> microcode : 0xa0b
>>> ...
>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm dtherm tpr_shadow vnmi flexpriority
>>>
>>> I bisected this back to:
>>>
>>> commit 34a1cd60d17f62c1f077c1478a6c2ca8c3d17af4
>>> Author: Tiejun Chen <[email protected]>
>>> Date: Tue Oct 28 10:14:48 2014 +0800
>>>
>>> kvm: x86: vmx: move some vmx setting from vmx_init() to hardware_setup()
>>>
>>> Instead of vmx_init(), actually it would make reasonable sense to do
>>> anything specific to vmx hardware setting in vmx_x86_ops->hardware_setup().
>>>
>>
>> This commit just reorders something but some MSR writing depend on previous
>> status.
>>
>> Could you try this?
>
> I unmangled the expanded tabs, and applied this:
>
> diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> index feb852b..96c84a8 100644
> --- a/arch/x86/kvm/vmx.c
> +++ b/arch/x86/kvm/vmx.c
> @@ -5840,49 +5840,6 @@ static __init int hardware_setup(void)
> memset(vmx_msr_bitmap_legacy, 0xff, PAGE_SIZE);
> memset(vmx_msr_bitmap_longmode, 0xff, PAGE_SIZE);
>
> - vmx_disable_intercept_for_msr(MSR_FS_BASE, false);
> - vmx_disable_intercept_for_msr(MSR_GS_BASE, false);
> - vmx_disable_intercept_for_msr(MSR_KERNEL_GS_BASE, true);
> - vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_CS, false);
> - vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_ESP, false);
> - vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_EIP, false);
> - vmx_disable_intercept_for_msr(MSR_IA32_BNDCFGS, true);
> -
> - memcpy(vmx_msr_bitmap_legacy_x2apic,
> - vmx_msr_bitmap_legacy, PAGE_SIZE);
> - memcpy(vmx_msr_bitmap_longmode_x2apic,
> - vmx_msr_bitmap_longmode, PAGE_SIZE);
> -
> - if (enable_apicv) {
> - for (msr = 0x800; msr <= 0x8ff; msr++)
> - vmx_disable_intercept_msr_read_x2apic(msr);
> -
> - /* According SDM, in x2apic mode, the whole id reg is used.
> - * But in KVM, it only use the highest eight bits. Need to
> - * intercept it */
> - vmx_enable_intercept_msr_read_x2apic(0x802);
> - /* TMCCT */
> - vmx_enable_intercept_msr_read_x2apic(0x839);
> - /* TPR */
> - vmx_disable_intercept_msr_write_x2apic(0x808);
> - /* EOI */
> - vmx_disable_intercept_msr_write_x2apic(0x80b);
> - /* SELF-IPI */
> - vmx_disable_intercept_msr_write_x2apic(0x83f);
> - }
> -
> - if (enable_ept) {
> - kvm_mmu_set_mask_ptes(0ull,
> - (enable_ept_ad_bits) ? VMX_EPT_ACCESS_BIT : 0ull,
> - (enable_ept_ad_bits) ? VMX_EPT_DIRTY_BIT : 0ull,
> - 0ull, VMX_EPT_EXECUTABLE_MASK);
> - ept_set_mmio_spte_mask();
> - kvm_enable_tdp();
> - } else
> - kvm_disable_tdp();
> -
> - update_ple_window_actual_max();
> -
> if (setup_vmcs_config(&vmcs_config) < 0) {
> r = -EIO;
> goto out7;
> @@ -5945,6 +5902,49 @@ static __init int hardware_setup(void)
> if (nested)
> nested_vmx_setup_ctls_msrs();
>
> + vmx_disable_intercept_for_msr(MSR_FS_BASE, false);
> + vmx_disable_intercept_for_msr(MSR_GS_BASE, false);
> + vmx_disable_intercept_for_msr(MSR_KERNEL_GS_BASE, true);
> + vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_CS, false);
> + vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_ESP, false);
> + vmx_disable_intercept_for_msr(MSR_IA32_SYSENTER_EIP, false);
> + vmx_disable_intercept_for_msr(MSR_IA32_BNDCFGS, true);
> +
> + memcpy(vmx_msr_bitmap_legacy_x2apic,
> + vmx_msr_bitmap_legacy, PAGE_SIZE);
> + memcpy(vmx_msr_bitmap_longmode_x2apic,
> + vmx_msr_bitmap_longmode, PAGE_SIZE);
> +
> + if (enable_apicv) {
> + for (msr = 0x800; msr <= 0x8ff; msr++)
> + vmx_disable_intercept_msr_read_x2apic(msr);
> +
> + /* According SDM, in x2apic mode, the whole id reg is used.
> + * But in KVM, it only use the highest eight bits. Need to
> + * intercept it */
> + vmx_enable_intercept_msr_read_x2apic(0x802);
> + /* TMCCT */
> + vmx_enable_intercept_msr_read_x2apic(0x839);
> + /* TPR */
> + vmx_disable_intercept_msr_write_x2apic(0x808);
> + /* EOI */
> + vmx_disable_intercept_msr_write_x2apic(0x80b);
> + /* SELF-IPI */
> + vmx_disable_intercept_msr_write_x2apic(0x83f);
> + }
> +
> + if (enable_ept) {
> + kvm_mmu_set_mask_ptes(0ull,
> + (enable_ept_ad_bits) ? VMX_EPT_ACCESS_BIT : 0ull,
> + (enable_ept_ad_bits) ? VMX_EPT_DIRTY_BIT : 0ull,
> + 0ull, VMX_EPT_EXECUTABLE_MASK);
> + ept_set_mmio_spte_mask();
> + kvm_enable_tdp();
> + } else
> + kvm_disable_tdp();
> +
> + update_ple_window_actual_max();
> +
> return alloc_kvm_area();
>
> out7:
>
>

2014-12-23 01:54:47

by Chen, Tiejun

Subject: Re: regression bisected; KVM: entry failed, hardware error 0x80000021

On 2014/12/23 9:50, Chen, Tiejun wrote:
> On 2014/12/22 17:23, Jamie Heilman wrote:
>> Chen, Tiejun wrote:
>>> On 2014/12/21 20:46, Jamie Heilman wrote:
>>>> With v3.19-rc1 when I run qemu-system-x86_64 -machine pc,accel=kvm I
>>>> get:
>>>>
>>>> KVM: entry failed, hardware error 0x80000021
>>>
>>> Looks some MSR writing issues such a failed entry.
>>>
>>>> If you're running a guest on an Intel machine without unrestricted mode
>>>> support, the failure can be most likely due to the guest entering an
>>>> invalid
>>>> state for Intel VT. For example, the guest maybe running in big real
>>>> mode
>>>> which is not supported on less recent Intel processors.
>>>>
>>>> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
>>>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
>>>> EIP=0000e05b EFL=00010002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>>> ES =0000 00000000 0000ffff 00009300
>>>> CS =f000 000f0000 0000ffff 00009b00
>>>> SS =0000 00000000 0000ffff 00009300
>>>> DS =0000 00000000 0000ffff 00009300
>>>> FS =0000 00000000 0000ffff 00009300
>>>> GS =0000 00000000 0000ffff 00009300
>>>> LDT=0000 00000000 0000ffff 00008200
>>>> TR =0000 00000000 0000ffff 00008b00
>>>> GDT= 00000000 0000ffff
>>>> IDT= 00000000 0000ffff
>>>> CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
>>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>>>> DR3=0000000000000000
>>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>>>> EFER=0000000000000000
>>>
>>> And I don't see any obvious wrong as well. Any valuable info from dmesg?
>>
>> With the simple qemu command above, on 3.18.1 I see:
>>
>> kern.info: kvm: zapping shadow pages for mmio generation wraparound
>>
>> when I fire up a full guest that's actually useful I get:
>>
>> kern.info: kvm: zapping shadow pages for mmio generation wraparound
>> kern.err: kvm [4073]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff
>>
>> On 3.18.0-rc3-00042-g34a1cd6 nothing appears in the dmesg, just the
>> message I mention above to stderr. Same thing with a stock
>> 3.19.0-rc1. Once I apply your patch the simple test command produces
>> the same zapping shadow pages messages as 3.18.1, and a test guest of
>> a Debian Jessie image (w/stock distro kernel) produces the same thing
>> with disabled perfctr wrmsr message. However, it doesn't look like
>
> Sorry I'm not sure if I understood current status. Looks 3.19-rc1 & my
> patch just fix that error above,
>
> KVM: entry failed, hardware error 0x80000021
> ...
>
> Right?
>
>> I'm entirely out of the woods, because one of my other guest VMs with a
>> custom kernel that works great under 3.18.1 now fails to run. Nothing
>> in dmesg, but here's the stderr:
>
> But even you revert 34a1cd60d17 or just apply my patch, something else
> introduced between 3.18.1 and 3.19-rc1 led this error below, right?
>
>>
>> KVM internal error. Suberror: 1
>> emulation failure
>> EAX=000de494 EBX=00000000 ECX=00000000 EDX=00000cfd
>> ESI=00000059 EDI=00000000 EBP=00000000 ESP=00006fb4
>> EIP=000f15c1 EFL=00010016 [----AP-] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
>> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>> DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>> FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>> GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>> LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
>> TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
>> GDT= 000f6be8 00000037
>> IDT= 000f6c26 00000000
>> CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>> DR3=0000000000000000
>> DR6=00000000ffff0ff0 DR7=0000000000000400
>> EFER=0000000000000000
>> Code=e8 ae fc ff ff 89 f2 a8 10 89 d8 75 0a b9 41 15 ff ff ff d1 <5b>
>> 5e c3 5b 5e e9 76 ff ff ff b0 11 e6 20 e6 a0 b0 08 e6 21 b0 70 e6 a1
>> b0 04 e6 21 b0 02
>>
>> FWIW, I get the same thing with 34a1cd60d17 reverted. Maybe there are
>> two bugs, maybe there's more to this first one. I can repro this
>
> So if my understanding is correct, this is probably another bug. And
> especially, I already saw the same log in another thread, "Cleaning up
> the KVM clock". Maybe you can continue to `git bisect` to locate that
> bad commit.
>

Looks just now Andy found that commit,
0e60b0799fedc495a5c57dbd669de3c10d72edd2 "kvm: change memslot sorting
rule from size to GFN", maybe you can try to revert this to try yours again.

Thanks
Tiejun

2014-12-23 07:27:03

by Jamie Heilman

Subject: Re: regression bisected; KVM: entry failed, hardware error 0x80000021

Chen, Tiejun wrote:
> On 2014/12/23 9:50, Chen, Tiejun wrote:
> >On 2014/12/22 17:23, Jamie Heilman wrote:
> >>Chen, Tiejun wrote:
> >>>On 2014/12/21 20:46, Jamie Heilman wrote:
> >>>>With v3.19-rc1 when I run qemu-system-x86_64 -machine pc,accel=kvm I
> >>>>get:
> >>>>
> >>>>KVM: entry failed, hardware error 0x80000021
> >>>
> >>>Looks some MSR writing issues such a failed entry.
> >>>
> >>>>If you're running a guest on an Intel machine without unrestricted mode
> >>>>support, the failure can be most likely due to the guest entering an
> >>>>invalid
> >>>>state for Intel VT. For example, the guest maybe running in big real
> >>>>mode
> >>>>which is not supported on less recent Intel processors.
> >>>>
> >>>>EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
> >>>>ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
> >>>>EIP=0000e05b EFL=00010002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> >>>>ES =0000 00000000 0000ffff 00009300
> >>>>CS =f000 000f0000 0000ffff 00009b00
> >>>>SS =0000 00000000 0000ffff 00009300
> >>>>DS =0000 00000000 0000ffff 00009300
> >>>>FS =0000 00000000 0000ffff 00009300
> >>>>GS =0000 00000000 0000ffff 00009300
> >>>>LDT=0000 00000000 0000ffff 00008200
> >>>>TR =0000 00000000 0000ffff 00008b00
> >>>>GDT= 00000000 0000ffff
> >>>>IDT= 00000000 0000ffff
> >>>>CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
> >>>>DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> >>>>DR3=0000000000000000
> >>>>DR6=00000000ffff0ff0 DR7=0000000000000400
> >>>>EFER=0000000000000000
> >>>
> >>>And I don't see any obvious wrong as well. Any valuable info from dmesg?
> >>
> >>With the simple qemu command above, on 3.18.1 I see:
> >>
> >>kern.info: kvm: zapping shadow pages for mmio generation wraparound
> >>
> >>when I fire up a full guest that's actually useful I get:
> >>
> >>kern.info: kvm: zapping shadow pages for mmio generation wraparound
> >>kern.err: kvm [4073]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff
> >>
> >>On 3.18.0-rc3-00042-g34a1cd6 nothing appears in the dmesg, just the
> >>message I mention above to stderr. Same thing with a stock
> >>3.19.0-rc1. Once I apply your patch the simple test command produces
> >>the same zapping shadow pages messages as 3.18.1, and a test guest of
> >>a Debian Jessie image (w/stock distro kernel) produces the same thing
> >>with disabled perfctr wrmsr message. However, it doesn't look like
> >
> >Sorry I'm not sure if I understood current status. Looks 3.19-rc1 & my
> >patch just fix that error above,
> >
> >KVM: entry failed, hardware error 0x80000021
> >...
> >
> >Right?
> >
> >>I'm entirely out of the woods, because one of my other guest VMs with a
> >>custom kernel that works great under 3.18.1 now fails to run. Nothing
> >>in dmesg, but here's the stderr:
> >
> >But even you revert 34a1cd60d17 or just apply my patch, something else
> >introduced between 3.18.1 and 3.19-rc1 led this error below, right?
> >
> >>
> >>KVM internal error. Suberror: 1
> >>emulation failure
> >>EAX=000de494 EBX=00000000 ECX=00000000 EDX=00000cfd
> >>ESI=00000059 EDI=00000000 EBP=00000000 ESP=00006fb4
> >>EIP=000f15c1 EFL=00010016 [----AP-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> >>ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> >>SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
> >>TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
> >>GDT= 000f6be8 00000037
> >>IDT= 000f6c26 00000000
> >>CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
> >>DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> >>DR3=0000000000000000
> >>DR6=00000000ffff0ff0 DR7=0000000000000400
> >>EFER=0000000000000000
> >>Code=e8 ae fc ff ff 89 f2 a8 10 89 d8 75 0a b9 41 15 ff ff ff d1 <5b>
> >>5e c3 5b 5e e9 76 ff ff ff b0 11 e6 20 e6 a0 b0 08 e6 21 b0 70 e6 a1
> >>b0 04 e6 21 b0 02
> >>
> >>FWIW, I get the same thing with 34a1cd60d17 reverted. Maybe there are
> >>two bugs, maybe there's more to this first one. I can repro this
> >
> >So if my understanding is correct, this is probably another bug. And
> >especially, I already saw the same log in another thread, "Cleaning up
> >the KVM clock". Maybe you can continue to `git bisect` to locate that
> >bad commit.
> >
>
> Looks just now Andy found that commit,
> 0e60b0799fedc495a5c57dbd669de3c10d72edd2 "kvm: change memslot sorting rule
> from size to GFN", maybe you can try to revert this to try yours again.

That doesn't revert cleanly for me, and I don't have much time to
fiddle with it until the 24th---so checked out the commit before it
(d4ae84a0), applied your patch, built, and yes, everything works fine
at that point. I'll probably have time for another full bisection
later, assuming things aren't ironed out already by then.

--
Jamie Heilman http://audible.transient.net/~jamie/

2014-12-23 07:51:00

by Chen, Tiejun

Subject: Re: regression bisected; KVM: entry failed, hardware error 0x80000021

On 2014/12/23 15:26, Jamie Heilman wrote:
> Chen, Tiejun wrote:
>> On 2014/12/23 9:50, Chen, Tiejun wrote:
>>> On 2014/12/22 17:23, Jamie Heilman wrote:
>>>> Chen, Tiejun wrote:
>>>>> On 2014/12/21 20:46, Jamie Heilman wrote:
>>>>>> With v3.19-rc1 when I run qemu-system-x86_64 -machine pc,accel=kvm I
>>>>>> get:
>>>>>>
>>>>>> KVM: entry failed, hardware error 0x80000021
>>>>>
>>>>> Looks some MSR writing issues such a failed entry.
>>>>>
>>>>>> If you're running a guest on an Intel machine without unrestricted mode
>>>>>> support, the failure can be most likely due to the guest entering an
>>>>>> invalid
>>>>>> state for Intel VT. For example, the guest maybe running in big real
>>>>>> mode
>>>>>> which is not supported on less recent Intel processors.
>>>>>>
>>>>>> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
>>>>>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
>>>>>> EIP=0000e05b EFL=00010002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>>>>> ES =0000 00000000 0000ffff 00009300
>>>>>> CS =f000 000f0000 0000ffff 00009b00
>>>>>> SS =0000 00000000 0000ffff 00009300
>>>>>> DS =0000 00000000 0000ffff 00009300
>>>>>> FS =0000 00000000 0000ffff 00009300
>>>>>> GS =0000 00000000 0000ffff 00009300
>>>>>> LDT=0000 00000000 0000ffff 00008200
>>>>>> TR =0000 00000000 0000ffff 00008b00
>>>>>> GDT= 00000000 0000ffff
>>>>>> IDT= 00000000 0000ffff
>>>>>> CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
>>>>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>>>>>> DR3=0000000000000000
>>>>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>>>>>> EFER=0000000000000000
>>>>>
>>>>> And I don't see any obvious wrong as well. Any valuable info from dmesg?
>>>>
>>>> With the simple qemu command above, on 3.18.1 I see:
>>>>
>>>> kern.info: kvm: zapping shadow pages for mmio generation wraparound
>>>>
>>>> when I fire up a full guest that's actually useful I get:
>>>>
>>>> kern.info: kvm: zapping shadow pages for mmio generation wraparound
>>>> kern.err: kvm [4073]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff
>>>>
>>>> On 3.18.0-rc3-00042-g34a1cd6 nothing appears in the dmesg, just the
>>>> message I mention above to stderr. Same thing with a stock
>>>> 3.19.0-rc1. Once I apply your patch the simple test command produces
>>>> the same zapping shadow pages messages as 3.18.1, and a test guest of
>>>> a Debian Jessie image (w/stock distro kernel) produces the same thing
>>>> with disabled perfctr wrmsr message. However, it doesn't look like
>>>
>>> Sorry I'm not sure if I understood current status. Looks 3.19-rc1 & my
>>> patch just fix that error above,
>>>
>>> KVM: entry failed, hardware error 0x80000021
>>> ...
>>>
>>> Right?
>>>
>>>> I'm entirely out of the woods, because one of my other guest VMs with a
>>>> custom kernel that works great under 3.18.1 now fails to run. Nothing
>>>> in dmesg, but here's the stderr:
>>>
>>> But even you revert 34a1cd60d17 or just apply my patch, something else
>>> introduced between 3.18.1 and 3.19-rc1 led this error below, right?
>>>
>>>>
>>>> KVM internal error. Suberror: 1
>>>> emulation failure
>>>> EAX=000de494 EBX=00000000 ECX=00000000 EDX=00000cfd
>>>> ESI=00000059 EDI=00000000 EBP=00000000 ESP=00006fb4
>>>> EIP=000f15c1 EFL=00010016 [----AP-] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>>> ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
>>>> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>> DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>> FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>> GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>> LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
>>>> TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
>>>> GDT= 000f6be8 00000037
>>>> IDT= 000f6c26 00000000
>>>> CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
>>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>>>> DR3=0000000000000000
>>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>>>> EFER=0000000000000000
>>>> Code=e8 ae fc ff ff 89 f2 a8 10 89 d8 75 0a b9 41 15 ff ff ff d1 <5b>
>>>> 5e c3 5b 5e e9 76 ff ff ff b0 11 e6 20 e6 a0 b0 08 e6 21 b0 70 e6 a1
>>>> b0 04 e6 21 b0 02
>>>>
>>>> FWIW, I get the same thing with 34a1cd60d17 reverted. Maybe there are
>>>> two bugs, maybe there's more to this first one. I can repro this
>>>
>>> So if my understanding is correct, this is probably another bug. And
>>> especially, I already saw the same log in another thread, "Cleaning up
>>> the KVM clock". Maybe you can continue to `git bisect` to locate that
>>> bad commit.
>>>
>>
>> Looks just now Andy found that commit,
>> 0e60b0799fedc495a5c57dbd669de3c10d72edd2 "kvm: change memslot sorting rule
>> from size to GFN", maybe you can try to revert this to try yours again.
>
> That doesn't revert cleanly for me, and I don't have much time to

Yeah, I guess all associated commits should be reverted gradually.

> fiddle with it until the 24th---so checked out the commit before it
> (d4ae84a0), applied your patch, built, and yes, everything works fine

Thanks for your test.

I think I can submit this patch to fix one of yours problems and I'd
like to add you as Reported-by & Tested-by.

Then we can step into another issue. And I'm trying to fetch 3.19-rc1
(because I'm always working on kvm/next.) to take a look at that but
maybe Paolo is already going on that.

Tiejun

> at that point. I'll probably have time for another full bisection
> later, assuming things aren't ironed out already by then.
>

2014-12-24 08:29:41

by Chen, Tiejun

Subject: Re: regression bisected; KVM: entry failed, hardware error 0x80000021

On 2014/12/23 15:26, Jamie Heilman wrote:
> Chen, Tiejun wrote:
>> On 2014/12/23 9:50, Chen, Tiejun wrote:
>>> On 2014/12/22 17:23, Jamie Heilman wrote:
>>>> Chen, Tiejun wrote:
>>>>> On 2014/12/21 20:46, Jamie Heilman wrote:
>>>>>> With v3.19-rc1 when I run qemu-system-x86_64 -machine pc,accel=kvm I
>>>>>> get:
>>>>>>
>>>>>> KVM: entry failed, hardware error 0x80000021
>>>>>
>>>>> Looks some MSR writing issues such a failed entry.
>>>>>
>>>>>> If you're running a guest on an Intel machine without unrestricted mode
>>>>>> support, the failure can be most likely due to the guest entering an
>>>>>> invalid
>>>>>> state for Intel VT. For example, the guest maybe running in big real
>>>>>> mode
>>>>>> which is not supported on less recent Intel processors.
>>>>>>
>>>>>> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
>>>>>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
>>>>>> EIP=0000e05b EFL=00010002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>>>>> ES =0000 00000000 0000ffff 00009300
>>>>>> CS =f000 000f0000 0000ffff 00009b00
>>>>>> SS =0000 00000000 0000ffff 00009300
>>>>>> DS =0000 00000000 0000ffff 00009300
>>>>>> FS =0000 00000000 0000ffff 00009300
>>>>>> GS =0000 00000000 0000ffff 00009300
>>>>>> LDT=0000 00000000 0000ffff 00008200
>>>>>> TR =0000 00000000 0000ffff 00008b00
>>>>>> GDT= 00000000 0000ffff
>>>>>> IDT= 00000000 0000ffff
>>>>>> CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
>>>>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>>>>>> DR3=0000000000000000
>>>>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>>>>>> EFER=0000000000000000
>>>>>
>>>>> And I don't see any obvious wrong as well. Any valuable info from dmesg?
>>>>
>>>> With the simple qemu command above, on 3.18.1 I see:
>>>>
>>>> kern.info: kvm: zapping shadow pages for mmio generation wraparound
>>>>
>>>> when I fire up a full guest that's actually useful I get:
>>>>
>>>> kern.info: kvm: zapping shadow pages for mmio generation wraparound
>>>> kern.err: kvm [4073]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff
>>>>
>>>> On 3.18.0-rc3-00042-g34a1cd6 nothing appears in the dmesg, just the
>>>> message I mention above to stderr. Same thing with a stock
>>>> 3.19.0-rc1. Once I apply your patch the simple test command produces
>>>> the same zapping shadow pages messages as 3.18.1, and a test guest of
>>>> a Debian Jessie image (w/stock distro kernel) produces the same thing
>>>> with disabled perfctr wrmsr message. However, it doesn't look like
>>>
>>> Sorry I'm not sure if I understood current status. Looks 3.19-rc1 & my
>>> patch just fix that error above,
>>>
>>> KVM: entry failed, hardware error 0x80000021
>>> ...
>>>
>>> Right?
>>>
>>>> I'm entirely out of the woods, because one of my other guest VMs with a
>>>> custom kernel that works great under 3.18.1 now fails to run. Nothing
>>>> in dmesg, but here's the stderr:
>>>
>>> But even you revert 34a1cd60d17 or just apply my patch, something else
>>> introduced between 3.18.1 and 3.19-rc1 led this error below, right?
>>>
>>>>
>>>> KVM internal error. Suberror: 1
>>>> emulation failure
>>>> EAX=000de494 EBX=00000000 ECX=00000000 EDX=00000cfd
>>>> ESI=00000059 EDI=00000000 EBP=00000000 ESP=00006fb4
>>>> EIP=000f15c1 EFL=00010016 [----AP-] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>>> ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
>>>> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>> DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>> FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>> GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>> LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
>>>> TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
>>>> GDT= 000f6be8 00000037
>>>> IDT= 000f6c26 00000000
>>>> CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
>>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>>>> DR3=0000000000000000
>>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>>>> EFER=0000000000000000
>>>> Code=e8 ae fc ff ff 89 f2 a8 10 89 d8 75 0a b9 41 15 ff ff ff d1 <5b>
>>>> 5e c3 5b 5e e9 76 ff ff ff b0 11 e6 20 e6 a0 b0 08 e6 21 b0 70 e6 a1
>>>> b0 04 e6 21 b0 02
>>>>
>>>> FWIW, I get the same thing with 34a1cd60d17 reverted. Maybe there are
>>>> two bugs, maybe there's more to this first one. I can repro this
>>>
>>> So if my understanding is correct, this is probably another bug. And
>>> especially, I already saw the same log in another thread, "Cleaning up
>>> the KVM clock". Maybe you can continue to `git bisect` to locate that
>>> bad commit.
>>>
>>
>> Looks just now Andy found that commit,
>> 0e60b0799fedc495a5c57dbd669de3c10d72edd2 "kvm: change memslot sorting rule
>> from size to GFN", maybe you can try to revert this to try yours again.
>
> That doesn't revert cleanly for me, and I don't have much time to
> fiddle with it until the 24th---so checked out the commit before it
> (d4ae84a0), applied your patch, built, and yes, everything works fine
> at that point. I'll probably have time for another full bisection
> later, assuming things aren't ironed out already by then.

Could you try this to fix your last error?

Signed-off-by: Tiejun Chen <[email protected]>
---
virt/kvm/kvm_main.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index f528343..a2d928c 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -672,6 +672,7 @@ static void update_memslots(struct kvm_memslots *slots,
WARN_ON(mslots[i].id != id);
if (!new->npages) {
new->base_gfn = 0;
+ new->flags = 0;
if (mslots[i].npages)
slots->used_slots--;
} else {
@@ -688,7 +689,7 @@ static void update_memslots(struct kvm_memslots *slots,
i++;
}
while (i > 0 &&
- new->base_gfn > mslots[i - 1].base_gfn) {
+ new->base_gfn >= mslots[i - 1].base_gfn) {
mslots[i] = mslots[i - 1];
slots->id_to_index[mslots[i].id] = i;
i--;
--
1.9.1

Tiejun

2014-12-24 11:02:44

by Jamie Heilman

Subject: Re: regression bisected; KVM: entry failed, hardware error 0x80000021

Chen, Tiejun wrote:
> On 2014/12/23 15:26, Jamie Heilman wrote:
> >Chen, Tiejun wrote:
> >>On 2014/12/23 9:50, Chen, Tiejun wrote:
> >>>On 2014/12/22 17:23, Jamie Heilman wrote:
> >>>>KVM internal error. Suberror: 1
> >>>>emulation failure
> >>>>EAX=000de494 EBX=00000000 ECX=00000000 EDX=00000cfd
> >>>>ESI=00000059 EDI=00000000 EBP=00000000 ESP=00006fb4
> >>>>EIP=000f15c1 EFL=00010016 [----AP-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> >>>>ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>>>CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> >>>>SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>>>DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>>>FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>>>GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>>>LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
> >>>>TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
> >>>>GDT= 000f6be8 00000037
> >>>>IDT= 000f6c26 00000000
> >>>>CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
> >>>>DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> >>>>DR3=0000000000000000
> >>>>DR6=00000000ffff0ff0 DR7=0000000000000400
> >>>>EFER=0000000000000000
> >>>>Code=e8 ae fc ff ff 89 f2 a8 10 89 d8 75 0a b9 41 15 ff ff ff d1 <5b>
> >>>>5e c3 5b 5e e9 76 ff ff ff b0 11 e6 20 e6 a0 b0 08 e6 21 b0 70 e6 a1
> >>>>b0 04 e6 21 b0 02
> >>>>
> >>>>FWIW, I get the same thing with 34a1cd60d17 reverted. Maybe there are
> >>>>two bugs, maybe there's more to this first one. I can repro this
> >>>
> >>>So if my understanding is correct, this is probably another bug. And
> >>>especially, I already saw the same log in another thread, "Cleaning up
> >>>the KVM clock". Maybe you can continue to `git bisect` to locate that
> >>>bad commit.
> >>>
> >>
> >>Looks just now Andy found that commit,
> >>0e60b0799fedc495a5c57dbd669de3c10d72edd2 "kvm: change memslot sorting rule
> >>from size to GFN", maybe you can try to revert this to try yours again.
> >
> >That doesn't revert cleanly for me, and I don't have much time to
> >fiddle with it until the 24th---so checked out the commit before it
> >(d4ae84a0), applied your patch, built, and yes, everything works fine
> >at that point. I'll probably have time for another full bisection
> >later, assuming things aren't ironed out already by then.

3.18.0-rc3-00120-gd4ae84a0 + vmx reorder msr writes patch = OK
3.18.0-rc3-00121-g0e60b07 + vmx reorder msr writes patch = emulation failure

So that certainly points to 0e60b0799fedc495a5c57dbd669de3c10d72edd2
as well.

> Could you try this to fix your last error?

Running qemu-system-x86_64 -machine pc,accel=kvm -nodefaults works,
my real (headless) kvm guests work, but this new patch makes running
"qemu-system-x86_64 -machine pc,accel=kvm" fail again, this time with
errors in the host to the tune of:

------------[ cut here ]------------
WARNING: CPU: 1 PID: 3901 at arch/x86/kvm/x86.c:6575 kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]()
Modules linked in: nfsv4 cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc vhost_net tun vhost macvtap macvlan fuse cbc dm_crypt usb_storage snd_hda_codec_analog snd_hda_codec_generic kvm_intel kvm tg3 ptp pps_core sr_mod snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer snd sg dcdbas cdrom psmouse soundcore floppy evdev xfs dm_mod raid1 md_mod
CPU: 1 PID: 3901 Comm: qemu-system-x86 Not tainted 3.19.0-rc1-00011-g53262d1-dirty #1
Hardware name: Dell Inc. Precision WorkStation T3400 /0TP412, BIOS A14 04/30/2012
0000000000000000 000000007e052328 ffff8800c25ffcf8 ffffffff813defbe
0000000000000000 0000000000000000 ffff8800c25ffd38 ffffffff8103b517
ffff8800c25ffd28 ffffffffa019bdec ffff8800caf1d000 ffff8800c2774800
Call Trace:
[<ffffffff813defbe>] dump_stack+0x4c/0x6e
[<ffffffff8103b517>] warn_slowpath_common+0x97/0xb1
[<ffffffffa019bdec>] ? kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
[<ffffffff8103b60b>] warn_slowpath_null+0x15/0x17
[<ffffffffa019bdec>] kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
[<ffffffffa02308b9>] ? vmcs_load+0x20/0x62 [kvm_intel]
[<ffffffffa0231e03>] ? vmx_vcpu_load+0x140/0x16a [kvm_intel]
[<ffffffffa0196ba3>] ? kvm_arch_vcpu_load+0x15c/0x161 [kvm]
[<ffffffffa018d8b1>] kvm_vcpu_ioctl+0x189/0x4bd [kvm]
[<ffffffff8104647a>] ? do_sigtimedwait+0x12f/0x189
[<ffffffff810ea316>] do_vfs_ioctl+0x370/0x436
[<ffffffff810f24f2>] ? __fget+0x67/0x72
[<ffffffff810ea41b>] SyS_ioctl+0x3f/0x5e
[<ffffffff813e34d2>] system_call_fastpath+0x12/0x17
---[ end trace 46abac932fb3b4a1 ]---
------------[ cut here ]------------
WARNING: CPU: 1 PID: 3901 at arch/x86/kvm/x86.c:6575 kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]()
Modules linked in: nfsv4 cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc vhost_net tun vhost macvtap macvlan fuse cbc dm_crypt usb_storage snd_hda_codec_analog snd_hda_codec_generic kvm_intel kvm tg3 ptp pps_core sr_mod snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer snd sg dcdbas cdrom psmouse soundcore floppy evdev xfs dm_mod raid1 md_mod
CPU: 1 PID: 3901 Comm: qemu-system-x86 Tainted: G W 3.19.0-rc1-00011-g53262d1-dirty #1
Hardware name: Dell Inc. Precision WorkStation T3400 /0TP412, BIOS A14 04/30/2012
0000000000000000 000000007e052328 ffff8800c25ffcf8 ffffffff813defbe
0000000000000000 0000000000000000 ffff8800c25ffd38 ffffffff8103b517
ffff8800c25ffd28 ffffffffa019bdec ffff8800caf1d000 ffff8800c2774800
Call Trace:
[<ffffffff813defbe>] dump_stack+0x4c/0x6e
[<ffffffff8103b517>] warn_slowpath_common+0x97/0xb1
[<ffffffffa019bdec>] ? kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
[<ffffffff8103b60b>] warn_slowpath_null+0x15/0x17
[<ffffffffa019bdec>] kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
[<ffffffffa02308b9>] ? vmcs_load+0x20/0x62 [kvm_intel]
[<ffffffffa0231e03>] ? vmx_vcpu_load+0x140/0x16a [kvm_intel]
[<ffffffffa0196ba3>] ? kvm_arch_vcpu_load+0x15c/0x161 [kvm]
[<ffffffffa018d8b1>] kvm_vcpu_ioctl+0x189/0x4bd [kvm]
[<ffffffff8104647a>] ? do_sigtimedwait+0x12f/0x189
[<ffffffff810ea316>] do_vfs_ioctl+0x370/0x436
[<ffffffff810f24f2>] ? __fget+0x67/0x72
[<ffffffff810ea41b>] SyS_ioctl+0x3f/0x5e
[<ffffffff813e34d2>] system_call_fastpath+0x12/0x17
---[ end trace 46abac932fb3b4a2 ]---

over and over and over ad nauseum, or until I kill the qemu command,
it also eats a core's worth of cpu.


--
Jamie Heilman http://audible.transient.net/~jamie/

2014-12-24 11:11:24

by Paolo Bonzini

Subject: Re: regression bisected; KVM: entry failed, hardware error 0x80000021

On 24/12/2014 12:02, Jamie Heilman wrote:
> Running qemu-system-x86_64 -machine pc,accel=kvm -nodefaults works,
> my real (headless) kvm guests work, but this new patch makes running
> "qemu-system-x86_64 -machine pc,accel=kvm" fail again, this time with
> errors in the host to the tune of:
>
> ------------[ cut here ]------------
> WARNING: CPU: 1 PID: 3901 at arch/x86/kvm/x86.c:6575 kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]()
> Modules linked in: nfsv4 cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc vhost_net tun vhost macvtap macvlan fuse cbc dm_crypt usb_storage snd_hda_codec_analog snd_hda_codec_generic kvm_intel kvm tg3 ptp pps_core sr_mod snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer snd sg dcdbas cdrom psmouse soundcore floppy evdev xfs dm_mod raid1 md_mod
> CPU: 1 PID: 3901 Comm: qemu-system-x86 Not tainted 3.19.0-rc1-00011-g53262d1-dirty #1
> Hardware name: Dell Inc. Precision WorkStation T3400 /0TP412, BIOS A14 04/30/2012
> 0000000000000000 000000007e052328 ffff8800c25ffcf8 ffffffff813defbe
> 0000000000000000 0000000000000000 ffff8800c25ffd38 ffffffff8103b517
> ffff8800c25ffd28 ffffffffa019bdec ffff8800caf1d000 ffff8800c2774800
> Call Trace:
> [<ffffffff813defbe>] dump_stack+0x4c/0x6e
> [<ffffffff8103b517>] warn_slowpath_common+0x97/0xb1
> [<ffffffffa019bdec>] ? kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
> [<ffffffff8103b60b>] warn_slowpath_null+0x15/0x17
> [<ffffffffa019bdec>] kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
> [<ffffffffa02308b9>] ? vmcs_load+0x20/0x62 [kvm_intel]
> [<ffffffffa0231e03>] ? vmx_vcpu_load+0x140/0x16a [kvm_intel]
> [<ffffffffa0196ba3>] ? kvm_arch_vcpu_load+0x15c/0x161 [kvm]
> [<ffffffffa018d8b1>] kvm_vcpu_ioctl+0x189/0x4bd [kvm]
> [<ffffffff8104647a>] ? do_sigtimedwait+0x12f/0x189
> [<ffffffff810ea316>] do_vfs_ioctl+0x370/0x436
> [<ffffffff810f24f2>] ? __fget+0x67/0x72
> [<ffffffff810ea41b>] SyS_ioctl+0x3f/0x5e
> [<ffffffff813e34d2>] system_call_fastpath+0x12/0x17
> ---[ end trace 46abac932fb3b4a1 ]---
> ------------[ cut here ]------------
> WARNING: CPU: 1 PID: 3901 at arch/x86/kvm/x86.c:6575 kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]()
> Modules linked in: nfsv4 cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc vhost_net tun vhost macvtap macvlan fuse cbc dm_crypt usb_storage snd_hda_codec_analog snd_hda_codec_generic kvm_intel kvm tg3 ptp pps_core sr_mod snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer snd sg dcdbas cdrom psmouse soundcore floppy evdev xfs dm_mod raid1 md_mod
> CPU: 1 PID: 3901 Comm: qemu-system-x86 Tainted: G W 3.19.0-rc1-00011-g53262d1-dirty #1
> Hardware name: Dell Inc. Precision WorkStation T3400 /0TP412, BIOS A14 04/30/2012
> 0000000000000000 000000007e052328 ffff8800c25ffcf8 ffffffff813defbe
> 0000000000000000 0000000000000000 ffff8800c25ffd38 ffffffff8103b517
> ffff8800c25ffd28 ffffffffa019bdec ffff8800caf1d000 ffff8800c2774800
> Call Trace:
> [<ffffffff813defbe>] dump_stack+0x4c/0x6e
> [<ffffffff8103b517>] warn_slowpath_common+0x97/0xb1
> [<ffffffffa019bdec>] ? kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
> [<ffffffff8103b60b>] warn_slowpath_null+0x15/0x17
> [<ffffffffa019bdec>] kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
> [<ffffffffa02308b9>] ? vmcs_load+0x20/0x62 [kvm_intel]
> [<ffffffffa0231e03>] ? vmx_vcpu_load+0x140/0x16a [kvm_intel]
> [<ffffffffa0196ba3>] ? kvm_arch_vcpu_load+0x15c/0x161 [kvm]
> [<ffffffffa018d8b1>] kvm_vcpu_ioctl+0x189/0x4bd [kvm]
> [<ffffffff8104647a>] ? do_sigtimedwait+0x12f/0x189
> [<ffffffff810ea316>] do_vfs_ioctl+0x370/0x436
> [<ffffffff810f24f2>] ? __fget+0x67/0x72
> [<ffffffff810ea41b>] SyS_ioctl+0x3f/0x5e
> [<ffffffff813e34d2>] system_call_fastpath+0x12/0x17
> ---[ end trace 46abac932fb3b4a2 ]---
>
> over and over and over ad nauseum, or until I kill the qemu command,
> it also eats a core's worth of cpu.

Yeah, I'm fairly sure that the second hunk of Tiejun's patch is not
correct, but he's on the right track. I hope to post a fix today, else
on the 27th or 29th.

Paolo

2014-12-25 07:46:38

by Chen, Tiejun

Subject: Re: regression bisected; KVM: entry failed, hardware error 0x80000021

On 2014/12/24 19:02, Jamie Heilman wrote:
> Chen, Tiejun wrote:
>> On 2014/12/23 15:26, Jamie Heilman wrote:
>>> Chen, Tiejun wrote:
>>>> On 2014/12/23 9:50, Chen, Tiejun wrote:
>>>>> On 2014/12/22 17:23, Jamie Heilman wrote:
>>>>>> KVM internal error. Suberror: 1
>>>>>> emulation failure
>>>>>> EAX=000de494 EBX=00000000 ECX=00000000 EDX=00000cfd
>>>>>> ESI=00000059 EDI=00000000 EBP=00000000 ESP=00006fb4
>>>>>> EIP=000f15c1 EFL=00010016 [----AP-] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>>>>> ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>>>> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
>>>>>> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>>>> DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>>>> FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>>>> GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>>>> LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
>>>>>> TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
>>>>>> GDT= 000f6be8 00000037
>>>>>> IDT= 000f6c26 00000000
>>>>>> CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
>>>>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>>>>>> DR3=0000000000000000
>>>>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>>>>>> EFER=0000000000000000
>>>>>> Code=e8 ae fc ff ff 89 f2 a8 10 89 d8 75 0a b9 41 15 ff ff ff d1 <5b>
>>>>>> 5e c3 5b 5e e9 76 ff ff ff b0 11 e6 20 e6 a0 b0 08 e6 21 b0 70 e6 a1
>>>>>> b0 04 e6 21 b0 02
>>>>>>
>>>>>> FWIW, I get the same thing with 34a1cd60d17 reverted. Maybe there are
>>>>>> two bugs, maybe there's more to this first one. I can repro this
>>>>>
>>>>> So if my understanding is correct, this is probably another bug. And
>>>>> especially, I already saw the same log in another thread, "Cleaning up
>>>>> the KVM clock". Maybe you can continue to `git bisect` to locate that
>>>>> bad commit.
>>>>>
>>>>
>>>> Looks just now Andy found that commit,
>>>> 0e60b0799fedc495a5c57dbd669de3c10d72edd2 "kvm: change memslot sorting rule
>>> >from size to GFN", maybe you can try to revert this to try yours again.
>>>
>>> That doesn't revert cleanly for me, and I don't have much time to
>>> fiddle with it until the 24th---so checked out the commit before it
>>> (d4ae84a0), applied your patch, built, and yes, everything works fine
>>> at that point. I'll probably have time for another full bisection
>>> later, assuming things aren't ironed out already by then.
>
> 3.18.0-rc3-00120-gd4ae84a0 + vmx reorder msr writes patch = OK
> 3.18.0-rc3-00121-g0e60b07 + vmx reorder msr writes patch = emulation failure
>
> So that certainly points to 0e60b0799fedc495a5c57dbd669de3c10d72edd2
> as well.
>
>> Could you try this to fix your last error?
>
> Running qemu-system-x86_64 -machine pc,accel=kvm -nodefaults works,
> my real (headless) kvm guests work, but this new patch makes running
> "qemu-system-x86_64 -machine pc,accel=kvm" fail again, this time with

Are you sure? From my test based on 3.19-rc1 that it owns top commit,

aa39477b5692611b91ac9455ae588738852b3f60

just plus my previous patch, "kvm: x86: vmx: reorder some msr writing"

I already can execute such a command successfully,

qemu-system-x86_64 -machine pc,accel=kvm -m 2048 -smp 2 -hda ubuntu.img

And your log below seems not to relate mem_slot issue we're discussing,
I guess you need to update qemu as well.

But I also found my new patch just work out Andy's next case, its really
bringing a new issue in !next case. So I tried to refine that patch
again as follows,

Signed-off-by: Tiejun Chen <[email protected]>
---
virt/kvm/kvm_main.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index f528343..910bc48 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -672,6 +672,7 @@ static void update_memslots(struct kvm_memslots *slots,
WARN_ON(mslots[i].id != id);
if (!new->npages) {
new->base_gfn = 0;
+ new->flags = 0;
if (mslots[i].npages)
slots->used_slots--;
} else {
@@ -688,7 +689,9 @@ static void update_memslots(struct kvm_memslots *slots,
i++;
}
while (i > 0 &&
- new->base_gfn > mslots[i - 1].base_gfn) {
+ ((new->base_gfn > mslots[i - 1].base_gfn) ||
+ (!new->base_gfn &&
+ !mslots[i - 1].base_gfn && !mslots[i - 1].npages))) {
mslots[i] = mslots[i - 1];
slots->id_to_index[mslots[i].id] = i;
i--;



Tiejun

> errors in the host to the tune of:
>
> ------------[ cut here ]------------
> WARNING: CPU: 1 PID: 3901 at arch/x86/kvm/x86.c:6575 kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]()
> Modules linked in: nfsv4 cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc vhost_net tun vhost macvtap macvlan fuse cbc dm_crypt usb_storage snd_hda_codec_analog snd_hda_codec_generic kvm_intel kvm tg3 ptp pps_core sr_mod snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer snd sg dcdbas cdrom psmouse soundcore floppy evdev xfs dm_mod raid1 md_mod
> CPU: 1 PID: 3901 Comm: qemu-system-x86 Not tainted 3.19.0-rc1-00011-g53262d1-dirty #1
> Hardware name: Dell Inc. Precision WorkStation T3400 /0TP412, BIOS A14 04/30/2012
> 0000000000000000 000000007e052328 ffff8800c25ffcf8 ffffffff813defbe
> 0000000000000000 0000000000000000 ffff8800c25ffd38 ffffffff8103b517
> ffff8800c25ffd28 ffffffffa019bdec ffff8800caf1d000 ffff8800c2774800
> Call Trace:
> [<ffffffff813defbe>] dump_stack+0x4c/0x6e
> [<ffffffff8103b517>] warn_slowpath_common+0x97/0xb1
> [<ffffffffa019bdec>] ? kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
> [<ffffffff8103b60b>] warn_slowpath_null+0x15/0x17
> [<ffffffffa019bdec>] kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
> [<ffffffffa02308b9>] ? vmcs_load+0x20/0x62 [kvm_intel]
> [<ffffffffa0231e03>] ? vmx_vcpu_load+0x140/0x16a [kvm_intel]
> [<ffffffffa0196ba3>] ? kvm_arch_vcpu_load+0x15c/0x161 [kvm]
> [<ffffffffa018d8b1>] kvm_vcpu_ioctl+0x189/0x4bd [kvm]
> [<ffffffff8104647a>] ? do_sigtimedwait+0x12f/0x189
> [<ffffffff810ea316>] do_vfs_ioctl+0x370/0x436
> [<ffffffff810f24f2>] ? __fget+0x67/0x72
> [<ffffffff810ea41b>] SyS_ioctl+0x3f/0x5e
> [<ffffffff813e34d2>] system_call_fastpath+0x12/0x17
> ---[ end trace 46abac932fb3b4a1 ]---
> ------------[ cut here ]------------
> WARNING: CPU: 1 PID: 3901 at arch/x86/kvm/x86.c:6575 kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]()
> Modules linked in: nfsv4 cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc vhost_net tun vhost macvtap macvlan fuse cbc dm_crypt usb_storage snd_hda_codec_analog snd_hda_codec_generic kvm_intel kvm tg3 ptp pps_core sr_mod snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer snd sg dcdbas cdrom psmouse soundcore floppy evdev xfs dm_mod raid1 md_mod
> CPU: 1 PID: 3901 Comm: qemu-system-x86 Tainted: G W 3.19.0-rc1-00011-g53262d1-dirty #1
> Hardware name: Dell Inc. Precision WorkStation T3400 /0TP412, BIOS A14 04/30/2012
> 0000000000000000 000000007e052328 ffff8800c25ffcf8 ffffffff813defbe
> 0000000000000000 0000000000000000 ffff8800c25ffd38 ffffffff8103b517
> ffff8800c25ffd28 ffffffffa019bdec ffff8800caf1d000 ffff8800c2774800
> Call Trace:
> [<ffffffff813defbe>] dump_stack+0x4c/0x6e
> [<ffffffff8103b517>] warn_slowpath_common+0x97/0xb1
> [<ffffffffa019bdec>] ? kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
> [<ffffffff8103b60b>] warn_slowpath_null+0x15/0x17
> [<ffffffffa019bdec>] kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
> [<ffffffffa02308b9>] ? vmcs_load+0x20/0x62 [kvm_intel]
> [<ffffffffa0231e03>] ? vmx_vcpu_load+0x140/0x16a [kvm_intel]
> [<ffffffffa0196ba3>] ? kvm_arch_vcpu_load+0x15c/0x161 [kvm]
> [<ffffffffa018d8b1>] kvm_vcpu_ioctl+0x189/0x4bd [kvm]
> [<ffffffff8104647a>] ? do_sigtimedwait+0x12f/0x189
> [<ffffffff810ea316>] do_vfs_ioctl+0x370/0x436
> [<ffffffff810f24f2>] ? __fget+0x67/0x72
> [<ffffffff810ea41b>] SyS_ioctl+0x3f/0x5e
> [<ffffffff813e34d2>] system_call_fastpath+0x12/0x17
> ---[ end trace 46abac932fb3b4a2 ]---
>
> over and over and over ad nauseum, or until I kill the qemu command,
> it also eats a core's worth of cpu.
>
>

2014-12-25 07:56:15

by Chen, Tiejun

Subject: Re: regression bisected; KVM: entry failed, hardware error 0x80000021

On 2014/12/24 19:11, Paolo Bonzini wrote:
>
>
> On 24/12/2014 12:02, Jamie Heilman wrote:
>> Running qemu-system-x86_64 -machine pc,accel=kvm -nodefaults works,
>> my real (headless) kvm guests work, but this new patch makes running
>> "qemu-system-x86_64 -machine pc,accel=kvm" fail again, this time with
>> errors in the host to the tune of:
>>
>> ------------[ cut here ]------------
>> WARNING: CPU: 1 PID: 3901 at arch/x86/kvm/x86.c:6575 kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]()
>> Modules linked in: nfsv4 cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc vhost_net tun vhost macvtap macvlan fuse cbc dm_crypt usb_storage snd_hda_codec_analog snd_hda_codec_generic kvm_intel kvm tg3 ptp pps_core sr_mod snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer snd sg dcdbas cdrom psmouse soundcore floppy evdev xfs dm_mod raid1 md_mod
>> CPU: 1 PID: 3901 Comm: qemu-system-x86 Not tainted 3.19.0-rc1-00011-g53262d1-dirty #1
>> Hardware name: Dell Inc. Precision WorkStation T3400 /0TP412, BIOS A14 04/30/2012
>> 0000000000000000 000000007e052328 ffff8800c25ffcf8 ffffffff813defbe
>> 0000000000000000 0000000000000000 ffff8800c25ffd38 ffffffff8103b517
>> ffff8800c25ffd28 ffffffffa019bdec ffff8800caf1d000 ffff8800c2774800
>> Call Trace:
>> [<ffffffff813defbe>] dump_stack+0x4c/0x6e
>> [<ffffffff8103b517>] warn_slowpath_common+0x97/0xb1
>> [<ffffffffa019bdec>] ? kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
>> [<ffffffff8103b60b>] warn_slowpath_null+0x15/0x17
>> [<ffffffffa019bdec>] kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
>> [<ffffffffa02308b9>] ? vmcs_load+0x20/0x62 [kvm_intel]
>> [<ffffffffa0231e03>] ? vmx_vcpu_load+0x140/0x16a [kvm_intel]
>> [<ffffffffa0196ba3>] ? kvm_arch_vcpu_load+0x15c/0x161 [kvm]
>> [<ffffffffa018d8b1>] kvm_vcpu_ioctl+0x189/0x4bd [kvm]
>> [<ffffffff8104647a>] ? do_sigtimedwait+0x12f/0x189
>> [<ffffffff810ea316>] do_vfs_ioctl+0x370/0x436
>> [<ffffffff810f24f2>] ? __fget+0x67/0x72
>> [<ffffffff810ea41b>] SyS_ioctl+0x3f/0x5e
>> [<ffffffff813e34d2>] system_call_fastpath+0x12/0x17
>> ---[ end trace 46abac932fb3b4a1 ]---
>> ------------[ cut here ]------------
>> WARNING: CPU: 1 PID: 3901 at arch/x86/kvm/x86.c:6575 kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]()
>> Modules linked in: nfsv4 cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc vhost_net tun vhost macvtap macvlan fuse cbc dm_crypt usb_storage snd_hda_codec_analog snd_hda_codec_generic kvm_intel kvm tg3 ptp pps_core sr_mod snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer snd sg dcdbas cdrom psmouse soundcore floppy evdev xfs dm_mod raid1 md_mod
>> CPU: 1 PID: 3901 Comm: qemu-system-x86 Tainted: G W 3.19.0-rc1-00011-g53262d1-dirty #1
>> Hardware name: Dell Inc. Precision WorkStation T3400 /0TP412, BIOS A14 04/30/2012
>> 0000000000000000 000000007e052328 ffff8800c25ffcf8 ffffffff813defbe
>> 0000000000000000 0000000000000000 ffff8800c25ffd38 ffffffff8103b517
>> ffff8800c25ffd28 ffffffffa019bdec ffff8800caf1d000 ffff8800c2774800
>> Call Trace:
>> [<ffffffff813defbe>] dump_stack+0x4c/0x6e
>> [<ffffffff8103b517>] warn_slowpath_common+0x97/0xb1
>> [<ffffffffa019bdec>] ? kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
>> [<ffffffff8103b60b>] warn_slowpath_null+0x15/0x17
>> [<ffffffffa019bdec>] kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
>> [<ffffffffa02308b9>] ? vmcs_load+0x20/0x62 [kvm_intel]
>> [<ffffffffa0231e03>] ? vmx_vcpu_load+0x140/0x16a [kvm_intel]
>> [<ffffffffa0196ba3>] ? kvm_arch_vcpu_load+0x15c/0x161 [kvm]
>> [<ffffffffa018d8b1>] kvm_vcpu_ioctl+0x189/0x4bd [kvm]
>> [<ffffffff8104647a>] ? do_sigtimedwait+0x12f/0x189
>> [<ffffffff810ea316>] do_vfs_ioctl+0x370/0x436
>> [<ffffffff810f24f2>] ? __fget+0x67/0x72
>> [<ffffffff810ea41b>] SyS_ioctl+0x3f/0x5e
>> [<ffffffff813e34d2>] system_call_fastpath+0x12/0x17
>> ---[ end trace 46abac932fb3b4a2 ]---
>>
>> over and over and over ad nauseum, or until I kill the qemu command,
>> it also eats a core's worth of cpu.

Such a message above seems to be out of our mem_slot issue, I'm not 100%
sure but actually I can run this case,

qemu-system-x86_64 -machine pc,accel=kvm -m 2048 -smp 2 -hda ubuntu.img

Just one patch, "kvm: x86: vmx: reorder some msr writing", is needed
here. So I guess you guy can try to pull your 3.19-rc1 + that patch, and
also pull qemu.

>
> Yeah, I'm fairly sure that the second hunk of Tiejun's patch is not
> correct, but he's on the right track. I hope to post a fix today, else

Yeah, looks that will broken !next case then I regenerate that again
post into another email. Now at lease I myself can run Andy's next case
and a normal case, "qemu-system-x86_64 -machine pc,accel=kvm", at the
same time. But if I'm missing something please correct me directly :)

Tiejun

2014-12-25 10:52:20

by Jamie Heilman

Subject: Re: regression bisected; KVM: entry failed, hardware error 0x80000021

Chen, Tiejun wrote:
> On 2014/12/24 19:02, Jamie Heilman wrote:
> >Chen, Tiejun wrote:
> >>On 2014/12/23 15:26, Jamie Heilman wrote:
> >>>Chen, Tiejun wrote:
> >>>>On 2014/12/23 9:50, Chen, Tiejun wrote:
> >>>>>On 2014/12/22 17:23, Jamie Heilman wrote:
> >>>>>>KVM internal error. Suberror: 1
> >>>>>>emulation failure
> >>>>>>EAX=000de494 EBX=00000000 ECX=00000000 EDX=00000cfd
> >>>>>>ESI=00000059 EDI=00000000 EBP=00000000 ESP=00006fb4
> >>>>>>EIP=000f15c1 EFL=00010016 [----AP-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> >>>>>>ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>>>>>CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> >>>>>>SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>>>>>DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>>>>>FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>>>>>GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
> >>>>>>LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
> >>>>>>TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
> >>>>>>GDT= 000f6be8 00000037
> >>>>>>IDT= 000f6c26 00000000
> >>>>>>CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
> >>>>>>DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> >>>>>>DR3=0000000000000000
> >>>>>>DR6=00000000ffff0ff0 DR7=0000000000000400
> >>>>>>EFER=0000000000000000
> >>>>>>Code=e8 ae fc ff ff 89 f2 a8 10 89 d8 75 0a b9 41 15 ff ff ff d1 <5b>
> >>>>>>5e c3 5b 5e e9 76 ff ff ff b0 11 e6 20 e6 a0 b0 08 e6 21 b0 70 e6 a1
> >>>>>>b0 04 e6 21 b0 02
> >>>>>>
> >>>>>>FWIW, I get the same thing with 34a1cd60d17 reverted. Maybe there are
> >>>>>>two bugs, maybe there's more to this first one. I can repro this
> >>>>>
> >>>>>So if my understanding is correct, this is probably another bug. And
> >>>>>especially, I already saw the same log in another thread, "Cleaning up
> >>>>>the KVM clock". Maybe you can continue to `git bisect` to locate that
> >>>>>bad commit.
> >>>>>
> >>>>
> >>>>Looks just now Andy found that commit,
> >>>>0e60b0799fedc495a5c57dbd669de3c10d72edd2 "kvm: change memslot sorting rule
> >>>>from size to GFN", maybe you can try to revert this to try yours again.
> >>>
> >>>That doesn't revert cleanly for me, and I don't have much time to
> >>>fiddle with it until the 24th---so checked out the commit before it
> >>>(d4ae84a0), applied your patch, built, and yes, everything works fine
> >>>at that point. I'll probably have time for another full bisection
> >>>later, assuming things aren't ironed out already by then.
> >
> >3.18.0-rc3-00120-gd4ae84a0 + vmx reorder msr writes patch = OK
> >3.18.0-rc3-00121-g0e60b07 + vmx reorder msr writes patch = emulation failure
> >
> >So that certainly points to 0e60b0799fedc495a5c57dbd669de3c10d72edd2
> >as well.
> >
> >>Could you try this to fix your last error?
> >
> >Running qemu-system-x86_64 -machine pc,accel=kvm -nodefaults works,
> >my real (headless) kvm guests work, but this new patch makes running
> >"qemu-system-x86_64 -machine pc,accel=kvm" fail again, this time with
>
> Are you sure? From my test based on 3.19-rc1 that it owns top commit,
>
> aa39477b5692611b91ac9455ae588738852b3f60
>
> just plus my previous patch, "kvm: x86: vmx: reorder some msr writing"
>
> I already can execute such a command successfully,
>
> qemu-system-x86_64 -machine pc,accel=kvm -m 2048 -smp 2 -hda ubuntu.img
>
> And your log below seems not to relate mem_slot issue we're discussing, I
> guess you need to update qemu as well.

Yes, I'm sure.

> But I also found my new patch just work out Andy's next case, its really
> bringing a new issue in !next case. So I tried to refine that patch again as
> follows,

This latest patch (again, after fixing all the whitespace so it actually
applies), does the trick. Both
"qemu-system-x86_64 -machine pc,accel=kvm" and
"qemu-system-x86_64 -machine pc,accel=kvm -nodefaults" work for me
now without any of the aforementioned warnings from the host.


> Signed-off-by: Tiejun Chen <[email protected]>
> ---
> virt/kvm/kvm_main.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index f528343..910bc48 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -672,6 +672,7 @@ static void update_memslots(struct kvm_memslots *slots,
> WARN_ON(mslots[i].id != id);
> if (!new->npages) {
> new->base_gfn = 0;
> + new->flags = 0;
> if (mslots[i].npages)
> slots->used_slots--;
> } else {
> @@ -688,7 +689,9 @@ static void update_memslots(struct kvm_memslots *slots,
> i++;
> }
> while (i > 0 &&
> - new->base_gfn > mslots[i - 1].base_gfn) {
> + ((new->base_gfn > mslots[i - 1].base_gfn) ||
> + (!new->base_gfn &&
> + !mslots[i - 1].base_gfn && !mslots[i - 1].npages))) {
> mslots[i] = mslots[i - 1];
> slots->id_to_index[mslots[i].id] = i;
> i--;
>
>
>
> Tiejun
>
> >errors in the host to the tune of:
> >
> >------------[ cut here ]------------
> >WARNING: CPU: 1 PID: 3901 at arch/x86/kvm/x86.c:6575 kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]()
> >Modules linked in: nfsv4 cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc vhost_net tun vhost macvtap macvlan fuse cbc dm_crypt usb_storage snd_hda_codec_analog snd_hda_codec_generic kvm_intel kvm tg3 ptp pps_core sr_mod snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer snd sg dcdbas cdrom psmouse soundcore floppy evdev xfs dm_mod raid1 md_mod
> >CPU: 1 PID: 3901 Comm: qemu-system-x86 Not tainted 3.19.0-rc1-00011-g53262d1-dirty #1
> >Hardware name: Dell Inc. Precision WorkStation T3400 /0TP412, BIOS A14 04/30/2012
> > 0000000000000000 000000007e052328 ffff8800c25ffcf8 ffffffff813defbe
> > 0000000000000000 0000000000000000 ffff8800c25ffd38 ffffffff8103b517
> > ffff8800c25ffd28 ffffffffa019bdec ffff8800caf1d000 ffff8800c2774800
> >Call Trace:
> > [<ffffffff813defbe>] dump_stack+0x4c/0x6e
> > [<ffffffff8103b517>] warn_slowpath_common+0x97/0xb1
> > [<ffffffffa019bdec>] ? kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
> > [<ffffffff8103b60b>] warn_slowpath_null+0x15/0x17
> > [<ffffffffa019bdec>] kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
> > [<ffffffffa02308b9>] ? vmcs_load+0x20/0x62 [kvm_intel]
> > [<ffffffffa0231e03>] ? vmx_vcpu_load+0x140/0x16a [kvm_intel]
> > [<ffffffffa0196ba3>] ? kvm_arch_vcpu_load+0x15c/0x161 [kvm]
> > [<ffffffffa018d8b1>] kvm_vcpu_ioctl+0x189/0x4bd [kvm]
> > [<ffffffff8104647a>] ? do_sigtimedwait+0x12f/0x189
> > [<ffffffff810ea316>] do_vfs_ioctl+0x370/0x436
> > [<ffffffff810f24f2>] ? __fget+0x67/0x72
> > [<ffffffff810ea41b>] SyS_ioctl+0x3f/0x5e
> > [<ffffffff813e34d2>] system_call_fastpath+0x12/0x17
> >---[ end trace 46abac932fb3b4a1 ]---
> >------------[ cut here ]------------
> >WARNING: CPU: 1 PID: 3901 at arch/x86/kvm/x86.c:6575 kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]()
> >Modules linked in: nfsv4 cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc vhost_net tun vhost macvtap macvlan fuse cbc dm_crypt usb_storage snd_hda_codec_analog snd_hda_codec_generic kvm_intel kvm tg3 ptp pps_core sr_mod snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer snd sg dcdbas cdrom psmouse soundcore floppy evdev xfs dm_mod raid1 md_mod
> >CPU: 1 PID: 3901 Comm: qemu-system-x86 Tainted: G W 3.19.0-rc1-00011-g53262d1-dirty #1
> >Hardware name: Dell Inc. Precision WorkStation T3400 /0TP412, BIOS A14 04/30/2012
> > 0000000000000000 000000007e052328 ffff8800c25ffcf8 ffffffff813defbe
> > 0000000000000000 0000000000000000 ffff8800c25ffd38 ffffffff8103b517
> > ffff8800c25ffd28 ffffffffa019bdec ffff8800caf1d000 ffff8800c2774800
> >Call Trace:
> > [<ffffffff813defbe>] dump_stack+0x4c/0x6e
> > [<ffffffff8103b517>] warn_slowpath_common+0x97/0xb1
> > [<ffffffffa019bdec>] ? kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
> > [<ffffffff8103b60b>] warn_slowpath_null+0x15/0x17
> > [<ffffffffa019bdec>] kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
> > [<ffffffffa02308b9>] ? vmcs_load+0x20/0x62 [kvm_intel]
> > [<ffffffffa0231e03>] ? vmx_vcpu_load+0x140/0x16a [kvm_intel]
> > [<ffffffffa0196ba3>] ? kvm_arch_vcpu_load+0x15c/0x161 [kvm]
> > [<ffffffffa018d8b1>] kvm_vcpu_ioctl+0x189/0x4bd [kvm]
> > [<ffffffff8104647a>] ? do_sigtimedwait+0x12f/0x189
> > [<ffffffff810ea316>] do_vfs_ioctl+0x370/0x436
> > [<ffffffff810f24f2>] ? __fget+0x67/0x72
> > [<ffffffff810ea41b>] SyS_ioctl+0x3f/0x5e
> > [<ffffffff813e34d2>] system_call_fastpath+0x12/0x17
> >---[ end trace 46abac932fb3b4a2 ]---
> >
> >over and over and over ad nauseum, or until I kill the qemu command,
> >it also eats a core's worth of cpu.

--
Jamie Heilman http://audible.transient.net/~jamie/

2014-12-26 01:44:33

by Chen, Tiejun

Subject: Re: regression bisected; KVM: entry failed, hardware error 0x80000021

On 2014/12/25 18:52, Jamie Heilman wrote:
> Chen, Tiejun wrote:
>> On 2014/12/24 19:02, Jamie Heilman wrote:
>>> Chen, Tiejun wrote:
>>>> On 2014/12/23 15:26, Jamie Heilman wrote:
>>>>> Chen, Tiejun wrote:
>>>>>> On 2014/12/23 9:50, Chen, Tiejun wrote:
>>>>>>> On 2014/12/22 17:23, Jamie Heilman wrote:
>>>>>>>> KVM internal error. Suberror: 1
>>>>>>>> emulation failure
>>>>>>>> EAX=000de494 EBX=00000000 ECX=00000000 EDX=00000cfd
>>>>>>>> ESI=00000059 EDI=00000000 EBP=00000000 ESP=00006fb4
>>>>>>>> EIP=000f15c1 EFL=00010016 [----AP-] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>>>>>>> ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>>>>>> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
>>>>>>>> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>>>>>> DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>>>>>> FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>>>>>> GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>>>>>>>> LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
>>>>>>>> TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
>>>>>>>> GDT= 000f6be8 00000037
>>>>>>>> IDT= 000f6c26 00000000
>>>>>>>> CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
>>>>>>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>>>>>>>> DR3=0000000000000000
>>>>>>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>>>>>>>> EFER=0000000000000000
>>>>>>>> Code=e8 ae fc ff ff 89 f2 a8 10 89 d8 75 0a b9 41 15 ff ff ff d1 <5b>
>>>>>>>> 5e c3 5b 5e e9 76 ff ff ff b0 11 e6 20 e6 a0 b0 08 e6 21 b0 70 e6 a1
>>>>>>>> b0 04 e6 21 b0 02
>>>>>>>>
>>>>>>>> FWIW, I get the same thing with 34a1cd60d17 reverted. Maybe there are
>>>>>>>> two bugs, maybe there's more to this first one. I can repro this
>>>>>>>
>>>>>>> So if my understanding is correct, this is probably another bug. And
>>>>>>> especially, I already saw the same log in another thread, "Cleaning up
>>>>>>> the KVM clock". Maybe you can continue to `git bisect` to locate that
>>>>>>> bad commit.
>>>>>>>
>>>>>>
>>>>>> Looks just now Andy found that commit,
>>>>>> 0e60b0799fedc495a5c57dbd669de3c10d72edd2 "kvm: change memslot sorting rule
>>>>> >from size to GFN", maybe you can try to revert this to try yours again.
>>>>>
>>>>> That doesn't revert cleanly for me, and I don't have much time to
>>>>> fiddle with it until the 24th---so checked out the commit before it
>>>>> (d4ae84a0), applied your patch, built, and yes, everything works fine
>>>>> at that point. I'll probably have time for another full bisection
>>>>> later, assuming things aren't ironed out already by then.
>>>
>>> 3.18.0-rc3-00120-gd4ae84a0 + vmx reorder msr writes patch = OK
>>> 3.18.0-rc3-00121-g0e60b07 + vmx reorder msr writes patch = emulation failure
>>>
>>> So that certainly points to 0e60b0799fedc495a5c57dbd669de3c10d72edd2
>>> as well.
>>>
>>>> Could you try this to fix your last error?
>>>
>>> Running qemu-system-x86_64 -machine pc,accel=kvm -nodefaults works,
>>> my real (headless) kvm guests work, but this new patch makes running
>>> "qemu-system-x86_64 -machine pc,accel=kvm" fail again, this time with
>>
>> Are you sure? From my test based on 3.19-rc1 that it owns top commit,
>>
>> aa39477b5692611b91ac9455ae588738852b3f60
>>
>> just plus my previous patch, "kvm: x86: vmx: reorder some msr writing"
>>
>> I already can execute such a command successfully,
>>
>> qemu-system-x86_64 -machine pc,accel=kvm -m 2048 -smp 2 -hda ubuntu.img
>>
>> And your log below seems not to relate mem_slot issue we're discussing, I
>> guess you need to update qemu as well.
>
> Yes, I'm sure.
>
>> But I also found my new patch just work out Andy's next case, its really
>> bringing a new issue in !next case. So I tried to refine that patch again as
>> follows,
>
> This latest patch (again, after fixing all the whitespace so it actually

Next time I guess I need to post that as a attached file :)

> applies), does the trick. Both
> "qemu-system-x86_64 -machine pc,accel=kvm" and
> "qemu-system-x86_64 -machine pc,accel=kvm -nodefaults" work for me
> now without any of the aforementioned warnings from the host.

Sounds great and thanks for your test again.

Tiejun

>
>
>> Signed-off-by: Tiejun Chen <[email protected]>
>> ---
>> virt/kvm/kvm_main.c | 5 ++++-
>> 1 file changed, 4 insertions(+), 1 deletion(-)
>>
>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>> index f528343..910bc48 100644
>> --- a/virt/kvm/kvm_main.c
>> +++ b/virt/kvm/kvm_main.c
>> @@ -672,6 +672,7 @@ static void update_memslots(struct kvm_memslots *slots,
>> WARN_ON(mslots[i].id != id);
>> if (!new->npages) {
>> new->base_gfn = 0;
>> + new->flags = 0;
>> if (mslots[i].npages)
>> slots->used_slots--;
>> } else {
>> @@ -688,7 +689,9 @@ static void update_memslots(struct kvm_memslots *slots,
>> i++;
>> }
>> while (i > 0 &&
>> - new->base_gfn > mslots[i - 1].base_gfn) {
>> + ((new->base_gfn > mslots[i - 1].base_gfn) ||
>> + (!new->base_gfn &&
>> + !mslots[i - 1].base_gfn && !mslots[i - 1].npages))) {
>> mslots[i] = mslots[i - 1];
>> slots->id_to_index[mslots[i].id] = i;
>> i--;
>>
>>
>>
>> Tiejun
>>
>>> errors in the host to the tune of:
>>>
>>> ------------[ cut here ]------------
>>> WARNING: CPU: 1 PID: 3901 at arch/x86/kvm/x86.c:6575 kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]()
>>> Modules linked in: nfsv4 cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc vhost_net tun vhost macvtap macvlan fuse cbc dm_crypt usb_storage snd_hda_codec_analog snd_hda_codec_generic kvm_intel kvm tg3 ptp pps_core sr_mod snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer snd sg dcdbas cdrom psmouse soundcore floppy evdev xfs dm_mod raid1 md_mod
>>> CPU: 1 PID: 3901 Comm: qemu-system-x86 Not tainted 3.19.0-rc1-00011-g53262d1-dirty #1
>>> Hardware name: Dell Inc. Precision WorkStation T3400 /0TP412, BIOS A14 04/30/2012
>>> 0000000000000000 000000007e052328 ffff8800c25ffcf8 ffffffff813defbe
>>> 0000000000000000 0000000000000000 ffff8800c25ffd38 ffffffff8103b517
>>> ffff8800c25ffd28 ffffffffa019bdec ffff8800caf1d000 ffff8800c2774800
>>> Call Trace:
>>> [<ffffffff813defbe>] dump_stack+0x4c/0x6e
>>> [<ffffffff8103b517>] warn_slowpath_common+0x97/0xb1
>>> [<ffffffffa019bdec>] ? kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
>>> [<ffffffff8103b60b>] warn_slowpath_null+0x15/0x17
>>> [<ffffffffa019bdec>] kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
>>> [<ffffffffa02308b9>] ? vmcs_load+0x20/0x62 [kvm_intel]
>>> [<ffffffffa0231e03>] ? vmx_vcpu_load+0x140/0x16a [kvm_intel]
>>> [<ffffffffa0196ba3>] ? kvm_arch_vcpu_load+0x15c/0x161 [kvm]
>>> [<ffffffffa018d8b1>] kvm_vcpu_ioctl+0x189/0x4bd [kvm]
>>> [<ffffffff8104647a>] ? do_sigtimedwait+0x12f/0x189
>>> [<ffffffff810ea316>] do_vfs_ioctl+0x370/0x436
>>> [<ffffffff810f24f2>] ? __fget+0x67/0x72
>>> [<ffffffff810ea41b>] SyS_ioctl+0x3f/0x5e
>>> [<ffffffff813e34d2>] system_call_fastpath+0x12/0x17
>>> ---[ end trace 46abac932fb3b4a1 ]---
>>> ------------[ cut here ]------------
>>> WARNING: CPU: 1 PID: 3901 at arch/x86/kvm/x86.c:6575 kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]()
>>> Modules linked in: nfsv4 cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_ondemand cpufreq_conservative autofs4 fan nfsd auth_rpcgss nfs lockd grace fscache sunrpc bridge stp llc vhost_net tun vhost macvtap macvlan fuse cbc dm_crypt usb_storage snd_hda_codec_analog snd_hda_codec_generic kvm_intel kvm tg3 ptp pps_core sr_mod snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer snd sg dcdbas cdrom psmouse soundcore floppy evdev xfs dm_mod raid1 md_mod
>>> CPU: 1 PID: 3901 Comm: qemu-system-x86 Tainted: G W 3.19.0-rc1-00011-g53262d1-dirty #1
>>> Hardware name: Dell Inc. Precision WorkStation T3400 /0TP412, BIOS A14 04/30/2012
>>> 0000000000000000 000000007e052328 ffff8800c25ffcf8 ffffffff813defbe
>>> 0000000000000000 0000000000000000 ffff8800c25ffd38 ffffffff8103b517
>>> ffff8800c25ffd28 ffffffffa019bdec ffff8800caf1d000 ffff8800c2774800
>>> Call Trace:
>>> [<ffffffff813defbe>] dump_stack+0x4c/0x6e
>>> [<ffffffff8103b517>] warn_slowpath_common+0x97/0xb1
>>> [<ffffffffa019bdec>] ? kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
>>> [<ffffffff8103b60b>] warn_slowpath_null+0x15/0x17
>>> [<ffffffffa019bdec>] kvm_arch_vcpu_ioctl_run+0xd63/0xe5b [kvm]
>>> [<ffffffffa02308b9>] ? vmcs_load+0x20/0x62 [kvm_intel]
>>> [<ffffffffa0231e03>] ? vmx_vcpu_load+0x140/0x16a [kvm_intel]
>>> [<ffffffffa0196ba3>] ? kvm_arch_vcpu_load+0x15c/0x161 [kvm]
>>> [<ffffffffa018d8b1>] kvm_vcpu_ioctl+0x189/0x4bd [kvm]
>>> [<ffffffff8104647a>] ? do_sigtimedwait+0x12f/0x189
>>> [<ffffffff810ea316>] do_vfs_ioctl+0x370/0x436
>>> [<ffffffff810f24f2>] ? __fget+0x67/0x72
>>> [<ffffffff810ea41b>] SyS_ioctl+0x3f/0x5e
>>> [<ffffffff813e34d2>] system_call_fastpath+0x12/0x17
>>> ---[ end trace 46abac932fb3b4a2 ]---
>>>
>>> over and over and over ad nauseum, or until I kill the qemu command,
>>> it also eats a core's worth of cpu.
>

2014-12-27 19:34:06

by Paolo Bonzini

Subject: Re: regression bisected; KVM: entry failed, hardware error 0x80000021

On 25/12/2014 08:46, Chen, Tiejun wrote:
>
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index f528343..910bc48 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -672,6 +672,7 @@ static void update_memslots(struct kvm_memslots *slots,
> WARN_ON(mslots[i].id != id);
> if (!new->npages) {
> new->base_gfn = 0;
> + new->flags = 0;
> if (mslots[i].npages)
> slots->used_slots--;
> } else {

Why is this assignment needed?

Paolo